Repository: netbox-community/netbox
Pull request: Fixes: #18021 - Clear Swagger/drf-spectacular API cache on startup
### Fixes: #18021

Clears the Swagger API schema cache upon startup, ensuring that API endpoint changes by developers are not cached and do not have to be manually refreshed upon every restart of the app.


Problem statement:
Swagger not updating when new API endpoints added in plugin
### Deployment Type

Self-hosted

### Triage priority

N/A

### NetBox Version

v4.1.3

### Python Version

3.11

### Steps to Reproduce

1) add an endpoint to an API plugin
2) reload netbox

### Expected Behavior

swagger should update showing the new endpoint

### Observed Behavior

Problem is caused by the caching of the swagger in redis, this needs to be cleared every time in order for the swagger to update. It does appear to load after the cache expires but this takes 24 hours

Hints from issue discussion:
Same problem with v4.1.6
Can this be addressed with documentation, e.g. a note that if you add an API endpoint in your plugin you need to flush your redis cache (with accompanying instructions)?
In my mind what should happen is the cached swagger should clear on container/app startup if its deemed out of date 
here a simple management command that fixes it once called `python manage.py $command_file_name`

$command_file_name.py

```
from django.core.management.commands.makemigrations import Command as _Command

from django.core.cache import cache


class Command(_Command):

    def handle(self, *args, **kwargs):
        """ Temporary command to clear cache keys that are not cleared by the cache clear command. """
        _KEYS_PREFIX_REQUIRING_CLEARING = [
            "*api_schema_*",
        ]
        for key_prefix in _KEYS_PREFIX_REQUIRING_CLEARING:
            matched_keys_to_delete = cache.keys(key_prefix)
            for matched_key in matched_keys_to_delete:
                cache.delete(matched_key)

```
This is a reminder that additional information is needed in order to further triage this issue. If the requested details are not provided, the issue will soon be closed automatically.
@bctiemann what information is missing here?
Seems like the ideal solution would be for NetBox to invalidate the cache automatically on startup, but what I'm not clear on is what the mechanism should be for it to determine that the Swagger schema is out of date. How should it "know" that anything has changed?

Alternatively, the manual cache-clearing script above would do the job, but I wonder if it would be more useful for it to just be a general "clear the entire cache" command, with an optional CLI arg to specify only the matching keys to delete (e.g. `*api_schema_*`), or else CLI args to target specific classes of keys, such as `--api_schema`. But this seems more like feature territory than a bug fix.

> Seems like the ideal solution would be for NetBox to invalidate the cache automatically on startup, but what I'm not clear on is what the mechanism should be for it to determine that the Swagger schema is out of date. How should it "know" that anything has changed?

I'm not a Django power user by any means, but what if instead of caching it in Redis across all instances, it was cached in-memory, per instance, during application startup? This would eliminate the need to bust the cache at all, would keep responses to requests quick and would presumably also clear during app restarts (like when using autoreload in development or when deploying new things).
@prryplatypus that actually sounds like how it should work -- but looking at the code now it seems like it's already written so as to cache based on a key that depends on the release version (e.g. `4.1.7`):

https://github.com/netbox-community/netbox/blob/020386f8a143126aee01b5481fbedb9bbc9efc33/netbox/netbox/urls.py#L57-L63

I think this handles the general use case of a user upgrading from one published version to another, but it doesn't address the case of a developer adding their own API endpoints within a version.

I think developers ought to be able to shoulder clearing the Redis cache manually in this scenario (`redis-cli flushall`). It seems reasonable to me to just document it accordingly (unless there's confirmation that the caching per-version isn't working?).

I realize (on rereading) that this is already generally known, but it feels like overkill to redesign the whole caching structure just for the sake of developers. (For that same reason I guess the management command approach suggested earlier wouldn't work either, as that would hardly be less work for the developer than just using `redis-cli flushall` each time they make a code change.)

Commit message: Clear Swagger API cache on startup

Patch to review:
diff --git a/netbox/core/apps.py b/netbox/core/apps.py
index 1dfc7a65ea3..c5051e92239 100644
--- a/netbox/core/apps.py
+++ b/netbox/core/apps.py
@@ -1,4 +1,5 @@
 from django.apps import AppConfig
+from django.core.cache import cache
 from django.db import models
 from django.db.migrations.operations import AlterModelOptions
 
@@ -22,3 +23,6 @@ def ready(self):
 
         # Register models
         register_models(*self.get_models())
+
+        # Clear Swagger API cache on startup
+        cache.delete("*api_schema_*")


Code review comments:
diff_hunk: @@ -22,3 +23,6 @@ def ready(self):
 
         # Register models
         register_models(*self.get_models())
+
+        # Clear Swagger API cache on startup
+        cache.delete("*api_schema_*")
path: netbox/core/apps.py
text: @user1:
This does not appear to work:

```
>>> cache.keys('*')
['views.decorators.cache.cache_header.api_schema_4.1.7.13748763de6e90ce226f69bcc045cbbb.en.UTC', 'dashboard_rss_628e2b38c2cec0d4aee7cbf72ba189194251c7fc25bd5e1210115aec39a705f4', 'views.decorators.cache.cache_page.api_schema_4.1.7.GET.13748763de6e90ce226f69bcc045cbbb.5c4af095dde9a1645c6d4c7f96a07693.en.UTC']
>>> cache.delete("*api_schema_*")
False
>>> cache.keys('*')
['views.decorators.cache.cache_header.api_schema_4.1.7.13748763de6e90ce226f69bcc045cbbb.en.UTC', 'dashboard_rss_628e2b38c2cec0d4aee7cbf72ba189194251c7fc25bd5e1210115aec39a705f4', 'views.decorators.cache.cache_page.api_schema_4.1.7.GET.13748763de6e90ce226f69bcc045cbbb.5c4af095dde9a1645c6d4c7f96a07693.en.UTC']
```

I don't think wildcard matching is supported. Deleting by the complete key name works:

```
>>> cache.delete("views.decorators.cache.cache_header.api_schema_4.1.7.13748763de6e90ce226f69bcc045cbbb.en.UTC")
True
```

@author:
Ah, thanks. I was going by the pattern from the script in the thread and presuming that had been verified to work.