Repository: netbox-community/netbox

Issue:
Showing config context with multiple tags assigned fails with MultipleObjectsReturned
### Environment
* Python version: 3.8.6
* NetBox version: 2.9.10

### Steps to Reproduce
1. create a virtual machine
2. add two tags (which result in adding data to config context)
3. Open Config context of that VM

<!-- What did you expect to happen? -->
### Expected Behavior
See config context

<!-- What happened instead? -->
### Observed Behavior
See an error
```
<class 'virtualization.models.VirtualMachine.MultipleObjectsReturned'>

get() returned more than one VirtualMachine -- it returned 2!
```
```
netbox_1         | Internal Server Error: /virtualization/virtual-machines/70/config-context/
netbox_1         | Traceback (most recent call last):
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
netbox_1         |     response = get_response(request)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 179, in _get_response
netbox_1         |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 73, in view
netbox_1         |     return self.dispatch(request, *args, **kwargs)
netbox_1         |   File "/opt/netbox/netbox/utilities/views.py", line 124, in dispatch
netbox_1         |     return super().dispatch(request, *args, **kwargs)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/views/generic/base.py", line 101, in dispatch
netbox_1         |     return handler(request, *args, **kwargs)
netbox_1         |   File "/opt/netbox/netbox/extras/views.py", line 146, in get
netbox_1         |     obj = get_object_or_404(self.queryset, pk=pk)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
netbox_1         |     return queryset.get(*args, **kwargs)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
netbox_1         |     return qs._no_monkey.get(qs, *args, **kwargs)
netbox_1         |   File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 433, in get
netbox_1         |     raise self.model.MultipleObjectsReturned(
netbox_1         | virtualization.models.VirtualMachine.MultipleObjectsReturned: get() returned more than one VirtualMachine -- it returned 2!
netbox_1         | 192.168.80.7 - - [29/Nov/2020:18:45:03 +0000] "GET /virtualization/virtual-machines/70/config-context/ HTTP/1.0" 500 1855 "-" "<cut>"
```

Note: I wrote this already in https://github.com/netbox-community/netbox/issues/5314#issuecomment-724722310 and [a change](https://github.com/netbox-community/netbox/commit/0d27abc6fc22a8d40183a59eceef5dda57e99eae) got introduced for 2.9 to fix it but in 2.10 it is still present.
I got asked to create a new issue.


Discussion:
I am also seeing this issue. In our case it is related to a device with two tags which provide config context data.

Exactly the same netbox version.

```Traceback (most recent call last):
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 47, in inner
   response = get_response(request)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 179, in _get_response
   response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/views/generic/base.py", line 73, in view
   return self.dispatch(request, *args, **kwargs)
 File "/opt/netbox/netbox/utilities/views.py", line 124, in dispatch
   return super().dispatch(request, *args, **kwargs)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/views/generic/base.py", line 101, in dispatch
   return handler(request, *args, **kwargs)
 File "/opt/netbox/netbox/extras/views.py", line 146, in get
   obj = get_object_or_404(self.queryset, pk=pk)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/shortcuts.py", line 76, in get_object_or_404
   return queryset.get(*args, **kwargs)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/cacheops/query.py", line 353, in get
   return qs._no_monkey.get(qs, *args, **kwargs)
 File "/opt/netbox/venv/lib64/python3.6/site-packages/django/db/models/query.py", line 436, in get
   num if not limit or num < limit else 'more than %s' % (limit - 1),
```
@lampwins can you take a look at this please?
It seems like the root issue here is that retrieving the tags assigned to a VM/device requires an `OUTER JOIN`, resulting in a replication of the VM/device for each tag that's assigned when the context data is applied. Here's an example query for a VM with three tags applied and a ConfigContext assigned to each tag:

```python
from django.db.models import OuterRef, Subquery, Q
from utilities.query_functions import EmptyGroupByJSONBAgg

qs = VirtualMachine.objects.annotate(
    config_context_data=Subquery(
        ConfigContext.objects.filter(
            Q(tags=OuterRef('tags')) | Q(tags=None)
        ).annotate(
            _data=EmptyGroupByJSONBAgg('data', ordering=['weight', 'name'])
        ).values("_data")
    )
).filter(id=5)
>>> print(qs.query)
SELECT "virtualization_virtualmachine"."id", "virtualization_virtualmachine"."created", "virtualization_virtualmachine"."last_updated", "virtualization_virtualmachine"."local_context_data", "virtualization_virtualmachine"."cluster_id", "virtualization_virtualmachine"."tenant_id", "virtualization_virtualmachine"."platform_id", "virtualization_virtualmachine"."name", "virtualization_virtualmachine"."status", "virtualization_virtualmachine"."role_id", "virtualization_virtualmachine"."primary_ip4_id", "virtualization_virtualmachine"."primary_ip6_id", "virtualization_virtualmachine"."vcpus", "virtualization_virtualmachine"."memory", "virtualization_virtualmachine"."disk", "virtualization_virtualmachine"."comments", (SELECT JSONB_AGG(U0."data" ORDER BY U0."weight", U0."name") AS "_data" FROM "extras_configcontext" U0 LEFT OUTER JOIN "extras_configcontext_tags" U1 ON (U0."id" = U1."configcontext_id") WHERE (U1."tag_id" = extras_taggeditem."tag_id" OR U1."tag_id" IS NULL)) AS "config_context_data" FROM "virtualization_virtualmachine" LEFT OUTER JOIN "extras_taggeditem" ON ("virtualization_virtualmachine"."id" = "extras_taggeditem"."object_id" AND ("extras_taggeditem"."content_type_id" = 72)) WHERE "virtualization_virtualmachine"."id" = 5 ORDER BY "virtualization_virtualmachine"."name" ASC, "virtualization_virtualmachine"."id" ASC
```

The SQL query (with some fields remove for brevity) returns the following:

```
netbox=> SELECT "virtualization_virtualmachine"."id", "virtualization_virtualmachine"."name", (SELECT JSONB_AGG(U0."data" ORDER BY U0."weight", U0."name") AS "_data" FROM "extras_configcontext" U0 LEFT OUTER JOIN "extras_configcontext_tags" U1 ON (U0."id" = U1."configcontext_id") WHERE (U1."tag_id" = extras_taggeditem."tag_id" OR U1."tag_id" IS NULL)) AS "config_context_data" FROM "virtualization_virtualmachine" LEFT OUTER JOIN "extras_taggeditem" ON ("virtualization_virtualmachine"."id" = "extras_taggeditem"."object_id" AND ("extras_taggeditem"."content_type_id" = 72)) WHERE "virtualization_virtualmachine"."id" = 5 ORDER BY "virtualization_virtualmachine"."name" ASC, "virtualization_virtualmachine"."id" ASC;
 id |   name   | config_context_data 
----+----------+---------------------
  5 | lab1-vm1 | [{"c": 3}]
  5 | lab1-vm1 | [{"b": 2}]
  5 | lab1-vm1 | [{"a": 1}]
(3 rows)
```

The best approach here isn't immediately clear. However, as this is a critical bug for users who employ config contexts, we'll have to disable the ConfigContextModelQuerySet optimization (introduced in #4559) in the next release if we can't identify a solution.
Would be happy to see a fix soon as we are heavily using config contexts for our device deployments.
I finally figured out why this did not come up when this was first raised in https://github.com/netbox-community/netbox/issues/5314. It turns out this happens in two situations and I only found the first one in the fix for #5314. The first instance is when a single config context has two tags assigned and a device/vm has both of those same two tags assigned. That case is actually covered in [this test case](https://github.com/netbox-community/netbox/blob/develop/netbox/extras/tests/test_models.py#L333). The second, outstanding case, is when a device/vm has two tags assigned and those two tags are assigned to two different config context objects.

Resolving patch:
diff --git a/netbox/extras/querysets.py b/netbox/extras/querysets.py
--- a/netbox/extras/querysets.py
+++ b/netbox/extras/querysets.py
@@ -2,6 +2,7 @@
 
 from django.db.models import OuterRef, Subquery, Q
 
+from extras.models.tags import TaggedItem
 from utilities.query_functions import EmptyGroupByJSONBAgg, OrderableJSONBAgg
 from utilities.querysets import RestrictedQuerySet
 
@@ -99,11 +100,25 @@ def annotate_config_context_data(self):
 
     def _get_config_context_filters(self):
         # Construct the set of Q objects for the specific object types
+        tag_query_filters = {
+            "object_id": OuterRef(OuterRef('pk')),
+            "content_type__app_label": self.model._meta.app_label,
+            "content_type__model": self.model._meta.model_name
+        }
         base_query = Q(
             Q(platforms=OuterRef('platform')) | Q(platforms=None),
             Q(tenant_groups=OuterRef('tenant__group')) | Q(tenant_groups=None),
             Q(tenants=OuterRef('tenant')) | Q(tenants=None),
-            Q(tags=OuterRef('tags')) | Q(tags=None),
+            Q(
+                tags__pk__in=Subquery(
+                    TaggedItem.objects.filter(
+                        **tag_query_filters
+                    ).values_list(
+                        'tag_id',
+                        flat=True
+                    )
+                )
+            ) | Q(tags=None),
             is_active=True,
         )
 


Test patch:
diff --git a/netbox/extras/tests/test_models.py b/netbox/extras/tests/test_models.py
--- a/netbox/extras/tests/test_models.py
+++ b/netbox/extras/tests/test_models.py
@@ -363,3 +363,46 @@ def test_multiple_tags_return_distinct_objects(self):
         annotated_queryset = Device.objects.filter(name=device.name).annotate_config_context_data()
         self.assertEqual(ConfigContext.objects.get_for_object(device).count(), 1)
         self.assertEqual(device.get_config_context(), annotated_queryset[0].get_config_context())
+
+    def test_multiple_tags_return_distinct_objects_with_seperate_config_contexts(self):
+        """
+        Tagged items use a generic relationship, which results in duplicate rows being returned when queried.
+        This is combatted by by appending distinct() to the config context querysets. This test creates a config
+        context assigned to two tags and ensures objects related by those same two tags result in only a single
+        config context record being returned.
+
+        This test case is seperate from the above in that it deals with multiple config context objects in play.
+
+        See https://github.com/netbox-community/netbox/issues/5387
+        """
+        tag_context_1 = ConfigContext.objects.create(
+            name="tag-1",
+            weight=100,
+            data={
+                "tag": 1
+            }
+        )
+        tag_context_1.tags.add(self.tag)
+        tag_context_2 = ConfigContext.objects.create(
+            name="tag-2",
+            weight=100,
+            data={
+                "tag": 1
+            }
+        )
+        tag_context_2.tags.add(self.tag2)
+
+        device = Device.objects.create(
+            name="Device 3",
+            site=self.site,
+            tenant=self.tenant,
+            platform=self.platform,
+            device_role=self.devicerole,
+            device_type=self.devicetype
+        )
+        device.tags.add(self.tag)
+        device.tags.add(self.tag2)
+
+        annotated_queryset = Device.objects.filter(name=device.name).annotate_config_context_data()
+        self.assertEqual(ConfigContext.objects.get_for_object(device).count(), 2)
+        self.assertEqual(device.get_config_context(), annotated_queryset[0].get_config_context())