Repository: nautobot/nautobot

Issue:
Provide health check for celery beat
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.6
* Nautobot version: 1.2.0b1

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->

We need to provide a healthcheck for Celery Beat.  In the docker-compose development file, we are using `nautobot-server health_check` for the Celery pods today.  Unfortunately, this health check doesn't check the health of the Celery worker or Celery beat.  The helm chart uses a slightly different approach for the celery worker which is:

```no-highlight
nautobot-server celery inspect ping --destination celery@$HOSTNAME
```

This works great for the Celery workers, but it doesn't work for Celery beat.  In some [quick research](https://thetldr.tech/how-to-perform-healthcheck-of-celery-beat/), we may need to implement a simple function to make this happen.  I wanted to get this open before we ship 1.2.


Discussion:
@nniehoff - Is there any update to this?

I've spotted that the health check applied to celery-beat in the Helm charts is `nautobot-server celery inspect ping --destination celery@$HOSTNAME` but won't work
Possibly something like this can help? https://github.com/celery/celery/issues/4079#issuecomment-1270085680

We are trialing this for the normal workers in a K8S environment
That looks to be worker-specific to me, and what we're looking for in this issue is a healthcheck for beat rather than the workers. Thanks though!
I thought that maybe there would be beat signals to cover the functionality, but it appears there is only [beat-init](https://docs.celeryq.dev/en/stable/userguide/signals.html#beat-init) and nothing after that.

Resolving patch:
diff --git a/nautobot/core/celery/schedulers.py b/nautobot/core/celery/schedulers.py
--- a/nautobot/core/celery/schedulers.py
+++ b/nautobot/core/celery/schedulers.py
@@ -1,7 +1,9 @@
 from collections.abc import Mapping
 import logging
+from pathlib import Path
 
 from celery import current_app
+from django.conf import settings
 from django_celery_beat.schedulers import DatabaseScheduler, ModelEntry
 from kombu.utils.json import loads
 
@@ -94,3 +96,14 @@ def apply_async(self, entry, producer=None, advance=True, **kwargs):
             entry.total_run_count = entry.model.total_run_count
             entry.model.save()
         return resp
+
+    def tick(self, *args, **kwargs):
+        """
+        Run a tick - one iteration of the scheduler.
+
+        This is an extension of `celery.beat.Scheduler.tick()` to touch the `CELERY_BEAT_HEARTBEAT_FILE` file.
+        """
+        interval = super().tick(*args, **kwargs)
+        if settings.CELERY_BEAT_HEARTBEAT_FILE:
+            Path(settings.CELERY_BEAT_HEARTBEAT_FILE).touch(exist_ok=True)
+        return interval
diff --git a/nautobot/core/settings.py b/nautobot/core/settings.py
--- a/nautobot/core/settings.py
+++ b/nautobot/core/settings.py
@@ -1,7 +1,9 @@
 import os
+import os.path
 import platform
 import re
 import sys
+import tempfile
 
 from django.contrib.messages import constants as messages
 import django.forms
@@ -767,6 +769,12 @@
 # Celery (used for background processing)
 #
 
+# Celery Beat heartbeat file path - will be touched by Beat each time it wakes up as a proof-of-health.
+CELERY_BEAT_HEARTBEAT_FILE = os.getenv(
+    "NAUTOBOT_CELERY_BEAT_HEARTBEAT_FILE",
+    os.path.join(tempfile.gettempdir(), "nautobot_celery_beat_heartbeat"),
+)
+
 # Celery broker URL used to tell workers where queues are located
 CELERY_BROKER_URL = os.getenv("NAUTOBOT_CELERY_BROKER_URL", parse_redis_connection(redis_database=0))
 


Test patch: