Repository: nautobot/nautobot

Issue:
invoke loaddata - traceback
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.8.5
* Nautobot version: 26b111640f43 (v1.0.3b1)

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
I haven't added custom status data in database. I also checked that the export file contains only a single entry with the primary key listed in traceback

```yaml
-   model: extras.status
    pk: 093f43ab-4d78-42ee-958f-76d86b1482a4
    fields:
        created: 2021-05-26
        last_updated: 2021-05-26 09:21:48.120080+00:00
        _custom_field_data: {}
        name: Retired
        color: f44336
        slug: retired
        description: Site has been retired
        content_types:
        - 2
```

### Steps to Reproduce
1. invoke dumpdata -f yaml
2. invoke destroy
3. invoke start
4. invoke createsuperuser
5. invoke loaddata -f db_output.yaml

<!-- What did you expect to happen? -->
### Expected Behavior
Data should be loaded

<!-- What happened instead? -->
### Observed Behavior
Traceback during load

```bash
 $ invoke loaddata -f db_output.yaml 
Running docker-compose command "ps --services --filter status=running"
Running docker-compose command "exec nautobot nautobot-server loaddata db_output.yaml"
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "extras_status_name_key"
DETAIL:  Key (name)=(Retired) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 5, in <module>
    main()
  File "/source/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/source/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: Problem installing fixture '/source/db_output.yaml': Could not load extras.Status(pk=093f43ab-4d78-42ee-958f-76d86b1482a4): duplicate key value violates unique constraint "extras_status_name_key"
DETAIL:  Key (name)=(Retired) already exists.
```


Discussion:
This is a common issue when importing from dumps because we're automatically creating `Status` objects when a new Nautobot database is bootstrapped.

Please try deleting all your `Status` objects on the new instance, prior to importing your dump:

```
$ nautobot-server nbshell
>>> Status.objects.all().delete()
```

And then run `nautobot-server loaddata`...
I still think it is reasonable for invoke loaddata to handle this for the user. I know I was surprised by this as well. 
If this is not handled by invoke loaddata then at least display a message to help resolving the problem.
A users expectation is that importing previously exported data would be possible without running into an error.
@jathanism  I've tested again with no success on import.

Python:

```bash
 $ invoke cli
Running docker-compose command "exec nautobot bash"
root@707eb9b5d878:/source# python --version
Python 3.8.7
```
Nautobot: 707eb9b5d878 (v1.0.4b1)

### First try

```bash
# delete all data

$ invoke destroy
Destroying Nautobot...
Running docker-compose command "down --volumes"
Stopping nautobot_worker_1   ... done
Stopping nautobot_nautobot_1 ... done
Stopping nautobot_redis_1    ... done
Stopping nautobot_postgres_1 ... done
Stopping nautobot_selenium_1 ... done
Removing nautobot_worker_1   ... done
Removing nautobot_nautobot_1 ... done
Removing nautobot_redis_1    ... done
Removing nautobot_postgres_1 ... done
Removing nautobot_selenium_1 ... done
Network nautobot-dev-net is external, skipping
Removing volume nautobot_pgdata_nautobot


# Startup Nautobot

$ invoke start
Starting Nautobot in detached mode...
Running docker-compose command "up --detach"
The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating volume "nautobot_pgdata_nautobot" with default driver
Creating nautobot_selenium_1 ... done
Creating nautobot_redis_1    ... done
Creating nautobot_postgres_1 ... done
Creating nautobot_nautobot_1 ... done
Creating nautobot_worker_1   ... done

# Successfully verified that WEB UI is responsive.

# try import - traceback

$ invoke loaddata db_output.json 
Running docker-compose command "ps --services --filter status=running"
Running docker-compose command "exec nautobot nautobot-server loaddata db_output.json"
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'status'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 146, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 270, in build_instance
    return Model(**data)
  File "/source/nautobot/dcim/models/cables.py", line 122, in __init__
    self._orig_status = self.status
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 429, in get
    raise self.model.DoesNotExist(
nautobot.extras.models.statuses.DoesNotExist: Status matching query does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 5, in <module>
    main()
  File "/source/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/source/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/source/db_output.json': 
```

### Second try

```bash
# delete all data

$ invoke destroy
Destroying Nautobot...
Running docker-compose command "down --volumes"
Stopping nautobot_worker_1   ... done
Stopping nautobot_nautobot_1 ... done
Stopping nautobot_postgres_1 ... done
Stopping nautobot_redis_1    ... done
Stopping nautobot_selenium_1 ... done
Removing nautobot_worker_1   ... done
Removing nautobot_nautobot_1 ... done
Removing nautobot_postgres_1 ... done
Removing nautobot_redis_1    ... done
Removing nautobot_selenium_1 ... done
Network nautobot-dev-net is external, skipping
Removing volume nautobot_pgdata_nautobot

# Start Nautobot

$ invoke start
Starting Nautobot in detached mode...
Running docker-compose command "up --detach"
The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating volume "nautobot_pgdata_nautobot" with default driver
Creating nautobot_selenium_1 ... done
Creating nautobot_redis_1    ... done
Creating nautobot_postgres_1 ... done
Creating nautobot_nautobot_1 ... done
Creating nautobot_worker_1   ... done

# Successfully verified than Web UI is responsive

# Delete Statuses from Database

$ invoke nbshell 
Running docker-compose command "ps --services --filter status=running"
Running docker-compose command "exec nautobot nautobot-server nbshell"
### Nautobot interactive shell (f77a7faadaf7)
### Python 3.8.7 | Django 3.1.12 | Nautobot 1.0.4b1
### lsmodels() will show available models. Use help(<model>) for more info.
>>> Status.objects.all().delete()
(67, {'extras.Status_content_types': 48, 'extras.Status': 19})
>>> quit()

# try import - traceback

$ invoke loaddata db_output.json 
Running docker-compose command "ps --services --filter status=running"
Running docker-compose command "exec nautobot nautobot-server loaddata db_output.json"
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 173, in __get__
    rel_obj = self.field.get_cached_value(instance)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/mixins.py", line 15, in get_cached_value
    return instance._state.fields_cache[cache_name]
KeyError: 'status'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    yield from PythonDeserializer(objects, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/python.py", line 146, in Deserializer
    obj = base.build_instance(Model, data, using)
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/base.py", line 270, in build_instance
    return Model(**data)
  File "/source/nautobot/dcim/models/cables.py", line 122, in __init__
    self._orig_status = self.status
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 187, in __get__
    rel_obj = self.get_object(instance)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/related_descriptors.py", line 154, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/usr/local/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 429, in get
    raise self.model.DoesNotExist(
nautobot.extras.models.statuses.DoesNotExist: Status matching query does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 5, in <module>
    main()
  File "/source/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/source/nautobot/core/runner/runner.py", line 266, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 172, in load_label
    for obj in objects:
  File "/usr/local/lib/python3.8/site-packages/django/core/serializers/json.py", line 74, in Deserializer
    raise DeserializationError() from exc
django.core.serializers.base.DeserializationError: Problem installing fixture '/source/db_output.json': 
```

@jathanism Hello, is it planned to fix this in near future or should I proceed using the database`s backup / restore functionality to transfer data between environments?

Resolving patch:
diff --git a/development/nautobot_config.py b/development/nautobot_config.py
--- a/development/nautobot_config.py
+++ b/development/nautobot_config.py
@@ -30,6 +30,10 @@
     }
 }
 
+# Ensure proper Unicode handling for MySQL
+if DATABASES["default"]["ENGINE"] == "django.db.backends.mysql":
+    DATABASES["default"]["OPTIONS"] = {"charset": "utf8mb4"}
+
 #
 # Debug
 #
diff --git a/nautobot/dcim/models/cables.py b/nautobot/dcim/models/cables.py
--- a/nautobot/dcim/models/cables.py
+++ b/nautobot/dcim/models/cables.py
@@ -118,8 +118,11 @@ def __init__(self, *args, **kwargs):
         # A copy of the PK to be used by __str__ in case the object is deleted
         self._pk = self.pk
 
-        # Cache the original status so we can check later if it's been changed
-        self._orig_status = self.status
+        if self.present_in_database:
+            # Cache the original status so we can check later if it's been changed
+            self._orig_status = self.status
+        else:
+            self._orig_status = None
 
     @classmethod
     def from_db(cls, db, field_names, values):
diff --git a/tasks.py b/tasks.py
--- a/tasks.py
+++ b/tasks.py
@@ -49,6 +49,7 @@ def is_truthy(arg):
             "compose_dir": os.path.join(os.path.dirname(__file__), "development/"),
             "compose_files": [
                 "docker-compose.yml",
+                "docker-compose.postgres.yml",
                 "docker-compose.dev.yml",
             ],
             # Image names to use when building from "main" branch


Test patch: