Repository: pulp/pulpcore

Issue:
Migrations must refuse to run if it is unsafe
## Motivation

The zero-downtime upgrade may require that all running pulp processes, i.e. workers, pulp-content, and pulp-api, are all at plugin or pulpcore version X or newer.

The zero-downtime group will ensure the database layer is backwards compatible by having db changes being only additive, but at some point a breaking db change must occur. For example if you want to rename a column, first add a new column, use both columns for a while, and then eventually remove the old column. The removal part of this example would be the breaking change.

## Proposal

The goal is to ensure migrations are always safe, and stop users from applying unsafe migrations.

* Breaking change migrations specify which versions of which components "must be" running before this change can be allowed
* Such migrations start by querying the DB for the current status (via heartbeat-msgs) of everything running in the system/cluster
* If anything is running at lower-than-required, the migration is halted
* In this situation, the "old" components are identified, and what-they-need-to-be-running is identified, and the admin is expected to upgrade/shut down any pods w/ old-code
* At that point, the upgrade can be restarted


Discussion:
This depends on #3175 

Resolving patch:
diff --git a/pulpcore/migrations.py b/pulpcore/migrations.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/migrations.py
@@ -0,0 +1,62 @@
+from packaging.version import parse as parse_version
+
+from django.conf import settings
+from django.utils import timezone
+from django.db.migrations.operations.base import Operation
+
+
+class RequireVersion(Operation):
+    reversible = True
+
+    def __init__(self, plugin, version, hints=None):
+        self.plugin = plugin
+        self.version = version
+        self.hints = hints or {}
+        self.elidable = True
+
+    def state_forwards(self, app_label, state):
+        pass
+
+    def database_forwards(self, app_label, schema_editor, from_state, to_state):
+        from_state.clear_delayed_apps_cache()
+        ApiAppStatus = from_state.apps.get_model("core", "ApiAppStatus")
+        ContentAppStatus = from_state.apps.get_model("core", "ContentAppStatus")
+        Worker = from_state.apps.get_model("core", "Worker")
+
+        needed_version = parse_version(self.version)
+        errors = []
+
+        for worker_class, ttl, class_name in [
+            (ApiAppStatus, settings.API_APP_TTL, "api server"),
+            (ContentAppStatus, settings.CONTENT_APP_TTL, "content server"),
+            (Worker, settings.WORKER_TTL, "pulp worker"),
+        ]:
+            for worker in worker_class.objects.filter(
+                last_heartbeat__gte=timezone.now() - timezone.timedelta(seconds=ttl)
+            ):
+                present_version = worker.versions.get(self.plugin)
+                if present_version is not None and parse_version(present_version) < needed_version:
+                    errors.append(
+                        f"  - '{self.plugin}'='{present_version}' "
+                        f"with {class_name} '{worker.name}'"
+                    )
+
+        if errors:
+            raise RuntimeError(
+                "\n".join(
+                    [
+                        "Incompatible versions detected "
+                        f"({self.plugin} >= {self.version} needed):",
+                        *errors,
+                        "Please shutdown or upgrade the outdated components before you "
+                        "continue the migration.",
+                    ]
+                )
+            )
+
+    def describe(self):
+        return f"Require plugin '{self.plugin}' >= '{self.version}'."
+
+    @property
+    def migration_name_fragment(self):
+        return f"require_{self.plugin}_{self.version}"
diff --git a/pulpcore/plugin/migrations.py b/pulpcore/plugin/migrations.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/plugin/migrations.py
@@ -0,0 +1,4 @@
+from pulpcore.migrations import RequireVersion
+
+
+__all__ = ["RequireVersion"]


Test patch: