Repository: nautobot/nautobot

Issue:
Concurrency issues(?) (with tasks workers?)
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.9
* Nautobot version: 1.1.0
* Redis: redis:6.2.5-alpine
* PSQL: postgres:13.3-alpine

Docker-compose extract:
```
services:
  nautobot: &nautobot
    image: networktocode/nautobot:1.1.0-py3.9
    depends_on:
      - postgres
      - redis
    volumes:
      - ./volumes/media:/opt/nautobot/media:z,rw
      - ./volumes/git:/opt/nautobot/git:z,rw
      - ./volumes/jobs:/opt/nautobot/jobs:z,rw
    environment:
      NAUTOBOT_DB_HOST: postgres
      NAUTOBOT_DB_USER: nautobot
      NAUTOBOT_DB_PASSWORD: nautobot
      NAUTOBOT_DB_NAME: nautobot
      NAUTOBOT_REDIS_HOST: redis
      NAUTOBOT_SECRET_KEY: "*****"
      NAUTOBOT_MAX_PAGE_SIZE: "50000"
      NAUTOBOT_CHANGELOG_RETENTION: "366"
      NAUTOBOT_METRICS_ENABLED: "true"
      #NAUTOBOT_CACHEOPS_ENABLED: "false"

  celery_worker:
    <<: *nautobot
    entrypoint: "nautobot-server celery worker -B -l INFO"
    networks:
      - default
    labels: []
    depends_on:
      - nautobot
    healthcheck:
      interval: 5s
      timeout: 5s
      start_period: 5s
      retries: 3
      test: ["CMD", "nautobot-server", "health_check"]

  rq_worker:
    <<: *nautobot
    entrypoint: "nautobot-server rqworker"
    networks:
      - default
    labels: []
    depends_on:
      - nautobot
    healthcheck:
      interval: 5s
      timeout: 5s
      start_period: 5s
      retries: 3
      test: ["CMD", "nautobot-server", "health_check"]

  # postgres - https://hub.docker.com/_/postgres
  postgres:
    image: postgres:13.3-alpine
    volumes:
      - ./volumes/pgsql:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: nautobot
      POSTGRES_PASSWORD: nautobot
      POSTGRES_DB: nautobot

  # redis - https://hub.docker.com/_/redis
  redis:
    image: redis:6.2.5-alpine
```

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
### Steps to Reproduce
1. Stop the Celery worker
2. Create jobs (doing many queries?):
```py
from nautobot.extras.jobs import Job
from nautobot.dcim.models import Device, Interface
import time


class TestA(Job):

  devices = Device.objects

  class Meta:
    read_only = True

  def test_a(self):
    j = 0
    for i, device in enumerate(self.devices.all()):
      for j, interface in enumerate(device.interfaces.all(), start=j):
        if j > 50:
          break
        self.log_info(obj=interface, message=f'Iteration {i}/{j}, name={device.name}/{interface.name}, {interface.connected_endpoint}')
        time.sleep(0.1)


class TestB(Job):

  devices = Device.objects

  class Meta:
    read_only = True

  def test_b(self):
    j = 0
    for i, device in enumerate(self.devices.all()):
      for j, interface in enumerate(device.interfaces.all(), start=j):
        if j > 50:
          break
        self.log_info(obj=interface, message=f'Iteration {i}/{j}, name={device.name}/{interface.name}, {interface.connected_endpoint}')
        time.sleep(0.1)
```

2. Start multiple instances of each (using the API if not allowed in the UI)
3. Start the worker

<!-- What did you expect to happen? -->
### Expected Behavior
Jobs are fully ran in the start order

<!-- What happened instead? -->
### Observed Behavior
Some jobs are stuck in either the pending or the running state, with errors in the worker logs

### Additional informations

This is the best repro I could find in many hours of really weird and random errors. I noticed setting `NAUTOBOT_CACHEOPS_ENABLED: 'false'` could help getting errors more often.

And in that case (no cacheops), right after an instance is started, listing the job results (which seems to imply `page_size` times git refresh ?) + loading another page (like job list or git list/detail) is also a good way to crash the web container with weird errors too, like:
* `<class 'AttributeError'>, 'datetime.date' object has no attribute 'encode'`
* `<class 'RuntimeError'>, generator raised StopIteration`
* `<class 'AttributeError'>, 'NoneType' object has no attribute 'DoesNotExist'`

And in the logs about the git refresh:
```
nautobot_1       | 18:42:23.628 INFO    nautobot.jobs :
nautobot_1       |   Repository successfully refreshed
nautobot_1       | 18:42:23.655 INFO    nautobot.jobs :
nautobot_1       |   Repository successfully refreshed
nautobot_1       | 18:42:23.677 INFO    nautobot.jobs :
nautobot_1       |   Repository successfully refreshed
[..]
```

So I'm not sure the issue is restricted to the jobs part.

Some more random errors observed:
* `<class 'django.contrib.contenttypes.models.ContentType.DoesNotExist'> ContentType matching query does not exist.`
* `<class 'django.db.utils.DatabaseError'> error with status PGRES_TUPLES_OK and no message from the libpq`
* `<class 'ValueError'> Field 'id' expected a number but got 'ryws2lq3ihs******md9auxf1ua3'.`

For the job part, a possible workaround seems to be to set the pool implementation to solo (https://docs.celeryproject.org/en/stable/reference/cli.html#cmdoption-celery-worker-P): 
```yaml
    entrypoint: "nautobot-server celery worker -B -l INFO-P solo"
```

This really feels like I'm doing something wrong (especially since nobody else seems complaining?), I really hope not, but if that's the case I can't point out what it is.

Best, Alexandre


Discussion:
This is an interesting one because it's kind of complex. We're going to need to spend a little time trying to reproduce this. Thanks again @u1735067 
Can confirm that with `CACHEOPS_ENABLED=False`, running multiple concurrent Jobs results in very odd errors on the Celery worker, such as:

* `TypeError: the JSON object must be str, bytes or bytearray, not 'UUID'`
* `nautobot.circuits.models.CircuitTermination.MultipleObjectsReturned: get() returned more than one CircuitTermination -- it returned more than 20!`
* `psycopg2.ProgrammingError: no results to fetch`

Resolving patch:
diff --git a/nautobot/core/celery.py b/nautobot/core/celery.py
--- a/nautobot/core/celery.py
+++ b/nautobot/core/celery.py
@@ -4,6 +4,7 @@
 import nautobot
 
 from celery import Celery, shared_task
+from celery.fixups.django import DjangoFixup
 from django.core.serializers.json import DjangoJSONEncoder
 from django.utils.module_loading import import_string
 from kombu.serialization import register
@@ -31,6 +32,11 @@
 #   should have a `CELERY_` prefix.
 app.config_from_object("django.conf:settings", namespace="CELERY")
 
+# Because of the chicken-and-egg Django settings bootstrapping issue,
+# Celery doesn't automatically install its Django-specific patches.
+# So we need to explicitly do so ourselves:
+DjangoFixup(app).install()
+
 # Load task modules from all registered Django apps.
 app.autodiscover_tasks()
 


Test patch: