Repository: nautobot/nautobot

Issue:
`nautobot-server loaddata` errors for ObjectChange records
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable): 34c06ba7a177 (v2.0.0b1)
* Python version: 3.8
* Database platform, version:
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. dumpdata as per Nautobot documentation (https://docs.nautobot.com/projects/core/en/v2.0.0-beta.1/administration/nautobot-server/?h=dumpdata#dumpdata)
2. loaddata as per Nautobot documentation
3.

<!-- What did you expect to happen? -->
### Expected Behavior

loaddata works
<!-- What happened instead? -->
### Observed Behavior

loaddata errors :
```
root@34c06ba7a177:/source# nautobot-server dumpdata --natural-foreign --natural-primary --exclude contenttypes --exclude auth.permission --format json --indent 2 --traceback  > /tmp/nautobot_dump.json
08:17:03.254 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
08:17:03.449 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositorySync
08:17:03.455 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositoryDryRun
08:17:04.736 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleDryRunJob
08:17:04.737 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJob
08:17:04.737 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleHiddenJob
08:17:04.738 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleLoggingJob
08:17:04.738 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJobHookReceiver
08:17:04.739 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleSimpleJobButtonReceiver
08:17:04.740 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleComplexJobButtonReceiver
```

```
root@34c06ba7a177:/source# nautobot-server loaddata --traceback /tmp/nautobot_dump.json
08:17:26.171 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
08:17:26.378 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositorySync
08:17:26.385 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositoryDryRun
08:17:28.808 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleDryRunJob
08:17:28.809 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJob
08:17:28.809 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleHiddenJob
08:17:28.810 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleLoggingJob
08:17:28.810 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJobHookReceiver
08:17:28.811 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleSimpleJobButtonReceiver
08:17:28.812 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleComplexJobButtonReceiver
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 148, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 263, in build_instance
    natural_key = Model(**data).natural_key()
  File "/source/nautobot/core/models/__init__.py", line 90, in natural_key
    for lookups in [lookup.split("__") for lookup in self.natural_key_field_lookups]:
  File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 61, in __get__
    return self.fget(cls)
  File "/source/nautobot/core/models/__init__.py", line 146, in natural_key_field_lookups
    raise AttributeError(
AttributeError: Unable to identify an intrinsic natural-key definition for ObjectChange. If there isn't at least one UniqueConstraint, unique_together, or field with unique=True, you probably need to explicitly declare the 'natural_key_field_names' for this model, or potentially override the default 'natural_key_field_lookups' implementation for this model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 6, in <module>
    sys.exit(main())
  File "/source/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/source/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 78, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 123, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/tmp/nautobot_dump.json':
```
`nautobot-server loaddata` errors for ObjectChange records
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Nautobot version (Docker tag too if applicable): 34c06ba7a177 (v2.0.0b1)
* Python version: 3.8
* Database platform, version:
* Middleware(s):

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. dumpdata as per Nautobot documentation (https://docs.nautobot.com/projects/core/en/v2.0.0-beta.1/administration/nautobot-server/?h=dumpdata#dumpdata)
2. loaddata as per Nautobot documentation
3.

<!-- What did you expect to happen? -->
### Expected Behavior

loaddata works
<!-- What happened instead? -->
### Observed Behavior

loaddata errors :
```
root@34c06ba7a177:/source# nautobot-server dumpdata --natural-foreign --natural-primary --exclude contenttypes --exclude auth.permission --format json --indent 2 --traceback  > /tmp/nautobot_dump.json
08:17:03.254 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
08:17:03.449 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositorySync
08:17:03.455 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositoryDryRun
08:17:04.736 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleDryRunJob
08:17:04.737 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJob
08:17:04.737 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleHiddenJob
08:17:04.738 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleLoggingJob
08:17:04.738 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJobHookReceiver
08:17:04.739 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleSimpleJobButtonReceiver
08:17:04.740 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleComplexJobButtonReceiver
```

```
root@34c06ba7a177:/source# nautobot-server loaddata --traceback /tmp/nautobot_dump.json
08:17:26.171 DEBUG   nautobot.core.celery __init__.py        import_jobs_as_celery_tasks() :
  Importing system Jobs
08:17:26.378 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositorySync
08:17:26.385 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job nautobot.core.jobs.GitRepositoryDryRun
08:17:28.808 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleDryRunJob
08:17:28.809 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJob
08:17:28.809 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleHiddenJob
08:17:28.810 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleLoggingJob
08:17:28.810 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleJobHookReceiver
08:17:28.811 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleSimpleJobButtonReceiver
08:17:28.812 DEBUG   nautobot.core.celery __init__.py                      register_jobs() :
  Registering job example_plugin.jobs.ExampleComplexJobButtonReceiver
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 148, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 263, in build_instance
    natural_key = Model(**data).natural_key()
  File "/source/nautobot/core/models/__init__.py", line 90, in natural_key
    for lookups in [lookup.split("__") for lookup in self.natural_key_field_lookups]:
  File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 61, in __get__
    return self.fget(cls)
  File "/source/nautobot/core/models/__init__.py", line 146, in natural_key_field_lookups
    raise AttributeError(
AttributeError: Unable to identify an intrinsic natural-key definition for ObjectChange. If there isn't at least one UniqueConstraint, unique_together, or field with unique=True, you probably need to explicitly declare the 'natural_key_field_names' for this model, or potentially override the default 'natural_key_field_lookups' implementation for this model.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 6, in <module>
    sys.exit(main())
  File "/source/nautobot/core/cli/__init__.py", line 54, in main
    run_app(
  File "/source/nautobot/core/runner/runner.py", line 263, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 78, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 123, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/tmp/nautobot_dump.json':
```


Discussion:
Are you dumping and loading between different Nautobot versions?
Looks like a miss in #1574, as ObjectChange isn't mentioned in #2907/#2908/#2909. More fuel for us needing to do #3752 though.
> Are you dumping and loading between different Nautobot versions?

Both on same container instance from  34c06ba7a177 (v2.0.0b1)
Are you dumping and loading between different Nautobot versions?
Looks like a miss in #1574, as ObjectChange isn't mentioned in #2907/#2908/#2909. More fuel for us needing to do #3752 though.
> Are you dumping and loading between different Nautobot versions?

Both on same container instance from  34c06ba7a177 (v2.0.0b1)

Resolving patch:
diff --git a/nautobot/extras/api/serializers.py b/nautobot/extras/api/serializers.py
--- a/nautobot/extras/api/serializers.py
+++ b/nautobot/extras/api/serializers.py
@@ -118,7 +118,8 @@ def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
 
         if not settings.CONFIG_CONTEXT_DYNAMIC_GROUPS_ENABLED:
-            self.fields.pop("dynamic_groups")
+            # In the case of a nested serializer, we won't have a `dynamic_groups` field at all.
+            self.fields.pop("dynamic_groups", None)
 
     class Meta:
         model = ConfigContext
diff --git a/nautobot/extras/migrations/0093_alter_objectchange_unique_together.py b/nautobot/extras/migrations/0093_alter_objectchange_unique_together.py
new file mode 100644
--- /dev/null
+++ b/nautobot/extras/migrations/0093_alter_objectchange_unique_together.py
@@ -0,0 +1,17 @@
+# Generated by Django 3.2.18 on 2023-06-12 13:55
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("contenttypes", "0002_remove_content_type_name"),
+        ("extras", "0092_uniqueness_fixup"),
+    ]
+
+    operations = [
+        migrations.AlterUniqueTogether(
+            name="objectchange",
+            unique_together={("time", "request_id", "changed_object_type", "changed_object_id")},
+        ),
+    ]
diff --git a/nautobot/extras/models/change_logging.py b/nautobot/extras/models/change_logging.py
--- a/nautobot/extras/models/change_logging.py
+++ b/nautobot/extras/models/change_logging.py
@@ -111,6 +111,12 @@ class ObjectChange(BaseModel):
     class Meta:
         ordering = ["-time"]
         get_latest_by = "time"
+        # [request_id, changed_object_type, changed_object_id] is not sufficient to uniquely identify an ObjectChange,
+        # as a single bulk-create or bulk-edit REST API request may modify the same object multiple times, such as in
+        # the case of creating CircuitTerminations for both ends of a single Circuit in a single request.
+        unique_together = [
+            ["time", "request_id", "changed_object_type", "changed_object_id"],
+        ]
         indexes = [
             models.Index(
                 name="extras_objectchange_triple_idx",


Test patch:
diff --git a/nautobot/extras/tests/test_api.py b/nautobot/extras/tests/test_api.py
--- a/nautobot/extras/tests/test_api.py
+++ b/nautobot/extras/tests/test_api.py
@@ -32,6 +32,8 @@
     DynamicGroupOperatorChoices,
     JobExecutionType,
     JobResultStatusChoices,
+    ObjectChangeActionChoices,
+    ObjectChangeEventContextChoices,
     RelationshipTypeChoices,
     SecretsGroupAccessTypeChoices,
     SecretsGroupSecretTypeChoices,
@@ -53,6 +55,7 @@
     JobLogEntry,
     JobResult,
     Note,
+    ObjectChange,
     Relationship,
     RelationshipAssociation,
     Role,
@@ -2367,6 +2370,69 @@ def setUpTestData(cls):
         )
 
 
+class ObjectChangeTest(APIViewTestCases.GetObjectViewTestCase, APIViewTestCases.ListObjectsViewTestCase):
+    model = ObjectChange
+
+    @classmethod
+    def setUpTestData(cls):
+        cc = ConfigContext.objects.create(name="Config Context 1", weight=100, data={"foo": 123})
+        cc_oc = cc.to_objectchange(ObjectChangeActionChoices.ACTION_CREATE)
+        cc_oc.request_id = uuid.uuid4()
+        cc_oc.change_context = ObjectChangeEventContextChoices.CONTEXT_WEB
+        cc_oc.change_context_detail = "extras:configcontext_edit"
+        cc_oc.validated_save()
+
+        location_oc = Location.objects.first().to_objectchange(ObjectChangeActionChoices.ACTION_UPDATE)
+        location_oc.request_id = uuid.uuid4()
+        location_oc.change_context = ObjectChangeEventContextChoices.CONTEXT_ORM
+        location_oc.validated_save()
+
+        git_oc = ObjectChange.objects.create(
+            user=None,
+            user_name="deleted",
+            request_id=cc_oc.request_id,
+            action=ObjectChangeActionChoices.ACTION_DELETE,
+            changed_object_type=ContentType.objects.get_for_model(GitRepository),
+            changed_object_id=uuid.UUID("7af2e8d5-6d53-4b79-b488-60448aaaa9e8"),
+            change_context=ObjectChangeEventContextChoices.CONTEXT_WEB,
+            change_context_detail="extras:gitrepository_delete",
+            related_object=cc_oc.changed_object,
+            object_repr="demo-git-datasource 2",
+            object_data={
+                "name": "demo-git-datasource 2",
+                "slug": "demo_git_datasource_2",
+                "tags": [],
+                "branch": "main",
+                "created": "2023-06-07T12:49:34.309Z",
+                "remote_url": "https://github.com/nautobot/demo-git-datasource.git",
+                "current_head": "94e88b76e87ccf1fdf48995d72ede86db4623d60",
+                "last_updated": "2023-06-07T12:49:36.368Z",
+                "custom_fields": {},
+                "secrets_group": None,
+                "provided_contents": ["extras.configcontext", "extras.configcontextschema", "extras.exporttemplate"],
+            },
+            object_data_v2={
+                "id": "7af2e8d5-6d53-4b79-b488-60448aaaa9e8",
+                "url": "/api/extras/git-repositories/7af2e8d5-6d53-4b79-b488-60448aaaa9e8/",
+                "name": "demo-git-datasource 2",
+                "slug": "demo_git_datasource_2",
+                "branch": "main",
+                "created": "2023-06-07T12:49:34.309312Z",
+                "display": "demo-git-datasource 2",
+                "notes_url": "/api/extras/git-repositories/7af2e8d5-6d53-4b79-b488-60448aaaa9e8/notes/",
+                "remote_url": "https://github.com/nautobot/demo-git-datasource.git",
+                "object_type": "extras.gitrepository",
+                "current_head": "94e88b76e87ccf1fdf48995d72ede86db4623d60",
+                "last_updated": "2023-06-07T12:49:36.368627Z",
+                "custom_fields": {},
+                "secrets_group": None,
+                "natural_key_slug": "demo-git-datasource+2",
+                "provided_contents": ["extras.configcontextschema", "extras.configcontext", "extras.exporttemplate"],
+            },
+        )
+        git_oc.validated_save()
+
+
 class RelationshipTest(APIViewTestCases.APIViewTestCase, RequiredRelationshipTestMixin):
     model = Relationship
 
diff --git a/nautobot/extras/tests/test_models.py b/nautobot/extras/tests/test_models.py
--- a/nautobot/extras/tests/test_models.py
+++ b/nautobot/extras/tests/test_models.py
@@ -1,6 +1,7 @@
 import os
 import tempfile
 from unittest import mock
+import uuid
 import warnings
 
 from django.conf import settings
@@ -24,7 +25,13 @@
     Platform,
 )
 from nautobot.extras.constants import JOB_OVERRIDABLE_FIELDS
-from nautobot.extras.choices import LogLevelChoices, SecretsGroupAccessTypeChoices, SecretsGroupSecretTypeChoices
+from nautobot.extras.choices import (
+    LogLevelChoices,
+    ObjectChangeActionChoices,
+    ObjectChangeEventContextChoices,
+    SecretsGroupAccessTypeChoices,
+    SecretsGroupSecretTypeChoices,
+)
 from nautobot.extras.jobs import get_job
 from nautobot.extras.models import (
     ComputedField,
@@ -38,6 +45,7 @@
     Job as JobModel,
     JobLogEntry,
     JobResult,
+    ObjectChange,
     Role,
     Secret,
     SecretsGroup,
@@ -906,6 +914,17 @@ def test_clean_input_validation(self):
         )
 
 
+class ObjectChangeTest(ModelTestCases.BaseModelTestCase):
+    model = ObjectChange
+
+    @classmethod
+    def setUpTestData(cls):
+        location_oc = Location.objects.first().to_objectchange(ObjectChangeActionChoices.ACTION_UPDATE)
+        location_oc.request_id = uuid.uuid4()
+        location_oc.change_context = ObjectChangeEventContextChoices.CONTEXT_ORM
+        location_oc.validated_save()
+
+
 class RoleTest(TestCase):
     """Tests for `Role` model class."""