Repository: kubearchive/kubearchive
Pull request #907: Introduce SQL migrations
Author: login: rh-hemartin
Labels: name: kind/feature
description: New Feature
name: area/database
description: Related to the Database
name: size/medium
description: Issues of a week of estimated work
Created: 2025-03-13T09:27:51Z | Merged: 2025-04-23T07:25:16Z

<!--  Thanks for sending a pull request!

Please read our contributor guidelines:
https://kubearchive.github.io/kubearchive/main/contributors/guide.html
-->

## Which Issue(s) this Pull Request Resolves
<!-- Please make sure to create an issue you can link to and mention it here
to automatically close it. If this PR covers part of the work, instead of
"Resolves #", use "Related to #"

Usage: `Resolves #<issue number>`, or `Resolves (paste link of issue)`.
-->

Resolves #789 

## Release Note
<!--
If this change has user-visible impact, write a release note in the block
below. Include the string "action required" if additional action is required of
users switching to the new release, for example in case of a breaking change.

Write as if you are speaking to users, not other KubeArchive contributors. If this
change has no user-visible impact, leave the code block as is.
-->

```release-note
Changed database setup to use migration files instead. Read the [installation documentation](https://kubearchive.github.io/kubearchive/main/getting-started/installation.html) for more information.
```

## Notes for Reviewers
<!--
Leave notes for the reviewers if you want to point their attention to some
specific place.
-->

<!--
Please label this pull request according to what type of issue you are addressing.
For reference on required PR/issue labels, read here:
https://kubearchive.github.io/kubearchive/main/contributors/release.html#_pull_request_labels

Add one of the following labels:
kind/bug
kind/documentation
kind/feature

Optionally add one or more of the following kinds if applicable:
kind/breaking
-->


Linked issue #789: Create a patch strategy for SQL
When KubeArchive enters into a more stable phase we want to offer upgrade paths from one version to the next. Our main idea is to keep one SQL to setup the DB in case of a new installation, then keep multiple "patches" to jump from version to version, with an initial one being the initialization of the DB we have on the first stable version.

The aim of this issue is to gather information, tools and opinions about this.

**Acceptance Criteria**

* Have a documentation page explaining how it works and how users should apply patches.
* Using the changes on our DB from our 0.x versions to build a test set of migrations.
Linked issue comments: user: login: maruiz93
created_at: 2025-01-28T11:03:50Z
body: I believe we can refer to this as a [schema migration](https://en.wikipedia.org/wiki/Schema_migration)

Although the linked page comments about the existence of zero downtime migration strategies I don't think that we need them. We can rely on users migrating the data to a second database with the new schema and deploy the new kubearchive version pointing to the new database if they have this zero downtime requirement.
Also, something I consider important for this process is to have the schema versioned. KubeArchive is versioned but the versions of the schema are expected to grow slower than the application version, so it should be a separate version number, and the version of kubearchive should come with a version of the schema.

The if a user wants to update from a version of kubearchive that has schema version 1, to the version of kubearchive that has the schema version 3, the user should know that it has to migrate its database to acomodate from version 1 to version 3 based on the different SQL that we provide for each version migration. In this case it should be a SQL to migrate from v1 to v2 and another to migrate from v2 to v3 so the user can apply both patches sequentially to achieve what they want.

One of the tools I know for this is [flyway](https://documentation.red-gate.com/flyway/getting-started-with-flyway), but we can take a look at others.
author_association: MEMBER
user: login: maruiz93
created_at: 2025-01-28T11:17:37Z
body: This is also this interesting tool: https://github.com/golang-migrate/migrate 

And it has some interesting [recommendations](https://github.com/golang-migrate/migrate/blob/master/MIGRATIONS.md) for writing migration files, for example: 
> Best practice for writing schema migration is that all migrations should be reversible

Even though the migration per se is applied by the kubearchive user and that as a tool we are talking about providing the SQL migration files, we need to test those and that's where these tools are important
author_association: MEMBER
user: login: rh-hemartin
created_at: 2025-02-12T15:12:20Z
body: See a first rough idea here https://github.com/rh-hemartin/kubearchive/blob/draft-sql-patches/SQL_PATCHES.md, you can download it and test it if you want. I'll be exposing the code in the community meeting
author_association: CONTRIBUTOR

Diff:
diff --git a/.github/workflows/go_test_integration.yml b/.github/workflows/go_test_integration.yml
index f370ca86e..834373e73 100644
--- a/.github/workflows/go_test_integration.yml
+++ b/.github/workflows/go_test_integration.yml
@@ -34,6 +34,12 @@ jobs:
           curl -Lo ./kubectl "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
           chmod +x ./kubectl
           sudo mv kubectl /usr/local/bin
+      - name: Install latest migrate
+        run: |
+          set -x
+          curl -L https://github.com/golang-migrate/migrate/releases/latest/download/migrate.linux-amd64.tar.gz | tar xvz migrate
+          chmod +x ./migrate
+          sudo mv migrate /usr/local/bin
       - name: Install KinD
         run: |
           set -x
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index 36b8824f6..bf7c99a0e 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -16,6 +16,7 @@ Install these tools:
 1. [`podman`](https://podman.io/docs/installation)
 1. [`kind`](https://kind.sigs.k8s.io/docs/user/quick-start/)
 1. [`yq`](https://github.com/mikefarah/yq/)
+1. [`migrate`](https://github.com/golang-migrate/migrate)
 
 ## Creating a fork repository
 
diff --git a/docs/modules/ROOT/nav.adoc b/docs/modules/ROOT/nav.adoc
index 6c25e36d1..ae3e2778e 100644
--- a/docs/modules/ROOT/nav.adoc
+++ b/docs/modules/ROOT/nav.adoc
@@ -10,6 +10,7 @@
 ** xref:configuration/enable-observability.adoc[Enable Observability]
 ** xref:configuration/kubearchive-logs.adoc[]
 ** xref:configuration/pprof.adoc[]
+** xref:configuration/upgrading.adoc[]
 
 * Contributors
 ** xref:contributors/guide.adoc[Guide]
@@ -29,3 +30,4 @@
 ** xref:design/decisions.adoc[]
 ** xref:design/logging-integration.adoc[]
 ** xref:design/database-interface.adoc[]
+** xref:design/database-migrations.adoc[]
diff --git a/docs/modules/ROOT/pages/configuration/upgrading.adoc b/docs/modules/ROOT/pages/configuration/upgrading.adoc
new file mode 100644
index 000000000..539cb5180
--- /dev/null
+++ b/docs/modules/ROOT/pages/configuration/upgrading.adoc
@@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails, the version of the database changes but it is left in a
+state with data and/or schema inconsistencies. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+When you fix the problem, force the migration to the desired version with
+link:https://github.com/golang-migrate/migrate/blob/master/GETTING_STARTED.md#forcing-your-database-version[the following command]
+(continuing the PostgreSQL example):
+
+[source,bash]
+----
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    force <VERSION>
+----
+
+[WARNING]
+====
+The schema version for `migrate force` must be:
+* `<VERSION -1>` when you rolled back the changes of `migrate` and you overrode the migration file.
+* `<VERSION>` when you manually completed the failed `migrate` execution.
+Then run `migrate up` again to make sure you apply the whole migration stack.
+====
diff --git a/docs/modules/ROOT/pages/design/database-migrations.adoc b/docs/modules/ROOT/pages/design/database-migrations.adoc
new file mode 100644
index 000000000..d33830be2
--- /dev/null
+++ b/docs/modules/ROOT/pages/design/database-migrations.adoc
@@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses runs the migrations to avoid permissions problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementations (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates part of the migration
+process but you can replace it if needed.
+
+Each database engine integration provides a `migrations` folder (`integrations/database/<db-name>/migrations`).
+All migrations have two files: `up` for upgrades and `down` for downgrades.
+Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
+
+== Execution
+
+=== Apply all migrations to get the schema up-to-date
+
+[source,bash]
+----
+migrate \
+    -path migrations/ \
+    -database <driver>://<username>:<password>@<ip/dns>:<port>/<database> \
+    up
+----
+
+=== Downgrade one version
+
+[source,bash]
+----
+migrate \
+    -path migrations/ \
+    -database <driver>://<username>:<password>@<ip/dns>:<port>/<database> \
+    down 1
+----
+
+For more details about the CLI usage of `migrate` see
+link:https://github.com/golang-migrate/migrate/tree/master/cmd/migrate[their CLI documentation].
diff --git a/docs/modules/ROOT/pages/getting-started/installation.adoc b/docs/modules/ROOT/pages/getting-started/installation.adoc
index 74daec570..38fed43d2 100644
--- a/docs/modules/ROOT/pages/getting-started/installation.adoc
+++ b/docs/modules/ROOT/pages/getting-started/installation.adoc
@@ -19,17 +19,24 @@ This document explains how to install and configure KubeArchive in your Kubernet
 
 == Prepare the Database
 
-. Create a database user `kubearchive`.
-. With a DB admin account, apply the schema provided in the [filename]`kubearchive.sql` file
-within the KubeArchive release:
+. Create a user and a password for KubeArchive to use.
+. Create a database for KubeArchive, with admin permissions granted to the new user.
+  Configure its timezone as UTC and its locale as UTF-8.
+. Clone the
+link:https://github.com/kubearchive/kubearchive[KubeArchive repository]
+and run the SQL migrations for your database, in this example PostgreSQL:
 +
 [source,bash,subs="attributes"]
 ----
-wget {download-path}/kubearchive.sql
-psql -h database.example.com \
--U admin \
--f kubearchive.sql
+git clone https://github.com/kubearchive/kubearchive.git --branch={page-component-display-version}
+cd kubearchive
+migrate \
+    -verbose -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
 ----
++
+This configures the DB with the proper schema so KubeArchive can write and read resources.
 
 More information about the schema and its customization in the
 xref:integrations/database.adoc#_configuration_and_customization[Database Integration]
diff --git a/integrations/database/postgresql/README.md b/integrations/database/postgresql/README.md
index 99a6ae7c3..a8309bdfc 100644
--- a/integrations/database/postgresql/README.md
+++ b/integrations/database/postgresql/README.md
@@ -1,17 +1,11 @@
 # Generating the KubeArchive schema file.
 
-Accessing the PostgreSQL database from outside the cluster requires port forwarding.  This
+Accessing the PostgreSQL database from outside the cluster requires port forwarding. This
 command should be run in its own terminal:
-```
+
+```bash
 kubectl -n postgresql port-forward service/kubearchive-rw 5433:5432 &
 ```
+
 The local port (5433) in this case can be changed, but whatever value is chosen
 needs to be used in the following command.
-
-The PostgreSQL `kubearchive.sql` schema file in this directory is automatically
-generated using the following command in a KubeArchive development environment:
-```
-pg_dump -h localhost -p 5433 -U kubearchive -C -c --if-exists -s
-```
-This file should always represent the current KubeArchive database schema. Upgrades will
-be handled separately.
diff --git a/integrations/database/postgresql/create-user.sql b/integrations/database/postgresql/create-user.sql
deleted file mode 100644
index e2f9a7da9..000000000
--- a/integrations/database/postgresql/create-user.sql
+++ /dev/null
@@ -1,2 +0,0 @@
-DROP ROLE IF EXISTS kubearchive;
-CREATE USER kubearchive WITH ENCRYPTED PASSWORD 'P0stgr3sdbP@ssword';
diff --git a/integrations/database/postgresql/install.sh b/integrations/database/postgresql/install.sh
index bdd04022a..e9f39e018 100755
--- a/integrations/database/postgresql/install.sh
+++ b/integrations/database/postgresql/install.sh
@@ -34,7 +34,10 @@ while ! nc -vz localhost ${LOCAL_PORT} > /dev/null 2>&1 ; do
     sleep 0.5
 done
 echo .
-psql -h localhost -U postgres -p ${LOCAL_PORT} -f kubearchive.sql
+psql -h localhost -U postgres -p ${LOCAL_PORT} -f setup.sql
+
+export PGPASSWORD="Databas3Passw0rd"  # notsecret
+migrate -verbose -path migrations/ -database postgresql://kubearchive:${PGPASSWORD}@localhost:${LOCAL_PORT}/kubearchive up
 
 # Kill all background jobs, including the port-forward started earlier.
 trap 'kill $(jobs -p)' EXIT
diff --git a/integrations/database/postgresql/kubearchive-user.yaml b/integrations/database/postgresql/kubearchive-user.yaml
deleted file mode 100644
index d7eaeef83..000000000
--- a/integrations/database/postgresql/kubearchive-user.yaml
+++ /dev/null
@@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
-# SPDX-License-Identifier: Apache-2.0
----
-apiVersion: v1
-kind: Secret
-metadata:
-  name: kubearchive-user
-type: kubernetes.io/basic-auth
-data:
-  host: a3ViZWFyY2hpdmUtcnc=
-  password: RGF0YWJhczNQYXNzdzByZA==  # notsecret
-  port: NTQzMg==
-  username: a3ViZWFyY2hpdmU=
diff --git a/integrations/database/postgresql/kubearchive.sql b/integrations/database/postgresql/kubearchive.sql
deleted file mode 100644
index b39480872..000000000
--- a/integrations/database/postgresql/kubearchive.sql
+++ /dev/null
@@ -1,269 +0,0 @@
---
--- PostgreSQL database dump
---
-
--- Dumped from database version 16.6 (Debian 16.6-1.pgdg110+1)
--- Dumped by pg_dump version 16.3
-
-SET statement_timeout = 0;
-SET lock_timeout = 0;
-SET idle_in_transaction_session_timeout = 0;
-SET client_encoding = 'UTF8';
-SET standard_conforming_strings = on;
-SELECT pg_catalog.set_config('search_path', '', false);
-SET check_function_bodies = false;
-SET xmloption = content;
-SET client_min_messages = warning;
-SET row_security = off;
-
-DROP DATABASE IF EXISTS kubearchive;
---
--- Name: kubearchive; Type: DATABASE; Schema: -; Owner: kubearchive
---
-
-CREATE DATABASE kubearchive WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.UTF-8';
-
-
-ALTER DATABASE kubearchive OWNER TO kubearchive;
-
-\connect kubearchive
-
-SET statement_timeout = 0;
-SET lock_timeout = 0;
-SET idle_in_transaction_session_timeout = 0;
-SET client_encoding = 'UTF8';
-SET standard_conforming_strings = on;
-SELECT pg_catalog.set_config('search_path', '', false);
-SET check_function_bodies = false;
-SET xmloption = content;
-SET client_min_messages = warning;
-SET row_security = off;
-
---
--- Name: kubearchive; Type: DATABASE PROPERTIES; Schema: -; Owner: kubearchive
---
-
-ALTER DATABASE kubearchive SET "TimeZone" TO 'UTC';
-ALTER DATABASE kubearchive SET work_mem TO '64MB';
-
-
-\connect kubearchive
-
-SET statement_timeout = 0;
-SET lock_timeout = 0;
-SET idle_in_transaction_session_timeout = 0;
-SET client_encoding = 'UTF8';
-SET standard_conforming_strings = on;
-SELECT pg_catalog.set_config('search_path', '', false);
-SET check_function_bodies = false;
-SET xmloption = content;
-SET client_min_messages = warning;
-SET row_security = off;
-
---
--- Name: trigger_set_timestamp(); Type: FUNCTION; Schema: public; Owner: kubearchive
---
-
-CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
-    LANGUAGE plpgsql
-    AS $$
-BEGIN
-	  NEW.updated_at = NOW();
-RETURN NEW;
-END;
-	$$;
-
-
-ALTER FUNCTION public.trigger_set_timestamp() OWNER TO kubearchive;
-
-SET default_tablespace = '';
-
-SET default_table_access_method = heap;
-
---
--- Name: log_url; Type: TABLE; Schema: public; Owner: kubearchive
---
-
-CREATE TABLE public.log_url (
-    id bigint NOT NULL,
-    uuid uuid NOT NULL,
-    url text NOT NULL,
-    container_name text NOT NULL,
-    json_path text,
-    created_at timestamp with time zone DEFAULT now() NOT NULL,
-    updated_at timestamp with time zone DEFAULT now() NOT NULL
-);
-
-
-ALTER TABLE public.log_url OWNER TO kubearchive;
-
---
--- Name: log_url_id_seq; Type: SEQUENCE; Schema: public; Owner: kubearchive
---
-
-CREATE SEQUENCE public.log_url_id_seq
-    START WITH 1
-    INCREMENT BY 1
-    NO MINVALUE
-    NO MAXVALUE
-    CACHE 1;
-
-
-ALTER SEQUENCE public.log_url_id_seq OWNER TO kubearchive;
-
---
--- Name: log_url_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: kubearchive
---
-
-ALTER SEQUENCE public.log_url_id_seq OWNED BY public.log_url.id;
-
-
---
--- Name: resource; Type: TABLE; Schema: public; Owner: kubearchive
---
-
-CREATE TABLE public.resource (
-    id bigint NOT NULL,
-    uuid uuid NOT NULL,
-    api_version character varying NOT NULL,
-    kind character varying NOT NULL,
-    name character varying NOT NULL,
-    namespace character varying NOT NULL,
-    resource_version character varying,
-    created_at timestamp with time zone DEFAULT now() NOT NULL,
-    updated_at timestamp with time zone DEFAULT now() NOT NULL,
-    cluster_updated_ts timestamp with time zone NOT NULL,
-    cluster_deleted_ts timestamp with time zone,
-    data jsonb NOT NULL
-);
-
-
-ALTER TABLE public.resource OWNER TO kubearchive;
-
---
--- Name: resource_id_seq; Type: SEQUENCE; Schema: public; Owner: kubearchive
---
-
-CREATE SEQUENCE public.resource_id_seq
-    START WITH 1
-    INCREMENT BY 1
-    NO MINVALUE
-    NO MAXVALUE
-    CACHE 1;
-
-
-ALTER SEQUENCE public.resource_id_seq OWNER TO kubearchive;
-
---
--- Name: resource_id_seq; Type: SEQUENCE OWNED BY; Schema: public; Owner: kubearchive
---
-
-ALTER SEQUENCE public.resource_id_seq OWNED BY public.resource.id;
-
-
---
--- Name: log_url id; Type: DEFAULT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.log_url ALTER COLUMN id SET DEFAULT nextval('public.log_url_id_seq'::regclass);
-
-
---
--- Name: resource id; Type: DEFAULT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.resource ALTER COLUMN id SET DEFAULT nextval('public.resource_id_seq'::regclass);
-
-
---
--- Name: log_url log_url_pkey; Type: CONSTRAINT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.log_url
-    ADD CONSTRAINT log_url_pkey PRIMARY KEY (id);
-
-
---
--- Name: resource resource_pkey; Type: CONSTRAINT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.resource
-    ADD CONSTRAINT resource_pkey PRIMARY KEY (id);
-
-
---
--- Name: resource resource_uuid_key; Type: CONSTRAINT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.resource
-    ADD CONSTRAINT resource_uuid_key UNIQUE (uuid);
-
-
---
--- Name: idx_creation_timestamp_id; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX idx_creation_timestamp_id ON public.resource USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
-
-
---
--- Name: idx_json_annotations; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX idx_json_annotations ON public.resource USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)));
-
-
---
--- Name: idx_json_labels; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX idx_json_labels ON public.resource USING gin ((((data -> 'metadata'::text) -> 'labels'::text)));
-
-
---
--- Name: idx_json_owners; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX idx_json_owners ON public.resource USING gin ((((data -> 'metadata'::text) -> 'ownerReferences'::text)) jsonb_path_ops);
-
-
---
--- Name: log_url_uuid_idx; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX log_url_uuid_idx ON public.resource USING btree (uuid);
-
-
---
--- Name: resource_kind_namespace_idx; Type: INDEX; Schema: public; Owner: kubearchive
---
-
-CREATE INDEX resource_kind_namespace_idx ON public.resource USING btree (kind, api_version, namespace);
-
-
---
--- Name: log_url set_timestamp; Type: TRIGGER; Schema: public; Owner: kubearchive
---
-
-CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
-
-
---
--- Name: resource set_timestamp; Type: TRIGGER; Schema: public; Owner: kubearchive
---
-
-CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
-
-
---
--- Name: log_url log_url_uuid_fkey; Type: FK CONSTRAINT; Schema: public; Owner: kubearchive
---
-
-ALTER TABLE ONLY public.log_url
-    ADD CONSTRAINT log_url_uuid_fkey FOREIGN KEY (uuid) REFERENCES public.resource(uuid) ON DELETE CASCADE;
-
-
---
--- PostgreSQL database dump complete
---
-
diff --git a/integrations/database/postgresql/migrations/01_initial_schema.down.sql b/integrations/database/postgresql/migrations/01_initial_schema.down.sql
new file mode 100644
index 000000000..9b42cdbe1
--- /dev/null
+++ b/integrations/database/postgresql/migrations/01_initial_schema.down.sql
@@ -0,0 +1,7 @@
+BEGIN;
+
+DROP TABLE public.log_url;
+DROP TABLE public.resource;
+DROP FUNCTION public.trigger_set_timestamp;
+
+COMMIT;
diff --git a/integrations/database/postgresql/migrations/01_initial_schema.up.sql b/integrations/database/postgresql/migrations/01_initial_schema.up.sql
new file mode 100644
index 000000000..5d42568dd
--- /dev/null
+++ b/integrations/database/postgresql/migrations/01_initial_schema.up.sql
@@ -0,0 +1,59 @@
+BEGIN;
+
+CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+BEGIN
+    NEW.updated_at = NOW();
+RETURN NEW;
+END;
+    $$;
+
+CREATE TABLE public.resource (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid UNIQUE NOT NULL,
+    api_version character varying NOT NULL,
+    kind character varying NOT NULL,
+    name character varying NOT NULL,
+    namespace character varying NOT NULL,
+    resource_version character varying,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL,
+    cluster_updated_ts timestamp with time zone NOT NULL,
+    cluster_deleted_ts timestamp with time zone,
+    data jsonb NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE TABLE public.log_url (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid NOT NULL REFERENCES public.resource(uuid) ON DELETE CASCADE,
+    url text NOT NULL,
+    container_name text NOT NULL,
+    json_path text,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE INDEX idx_creation_timestamp_id ON public.resource
+    USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
+
+CREATE INDEX idx_json_annotations ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)));
+
+CREATE INDEX idx_json_labels ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'labels'::text)));
+
+CREATE INDEX idx_json_owners ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'ownerReferences'::text)) jsonb_path_ops);
+
+CREATE INDEX log_url_uuid_idx ON public.resource
+    USING btree (uuid);
+
+CREATE INDEX resource_kind_namespace_idx ON public.resource
+    USING btree (kind, api_version, namespace);
+
+COMMIT;
diff --git a/integrations/database/postgresql/postgres.yaml b/integrations/database/postgresql/postgres.yaml
index c003da9e7..d7d9e69ba 100644
--- a/integrations/database/postgresql/postgres.yaml
+++ b/integrations/database/postgresql/postgres.yaml
@@ -12,12 +12,6 @@ spec:
     # columns: connection-type database user address auth-options
     pg_hba:
       - hostnossl  all  all  all  reject
-  bootstrap:
-    initdb:
-      database: kubearchive
-      secret:
-        name: kubearchive-user
-      owner: kubearchive
   enableSuperuserAccess: true
   imageName: ghcr.io/cloudnative-pg/postgresql:16.6
   storage:
diff --git a/integrations/database/postgresql/setup.sql b/integrations/database/postgresql/setup.sql
new file mode 100644
index 000000000..ff500bb6a
--- /dev/null
+++ b/integrations/database/postgresql/setup.sql
@@ -0,0 +1,6 @@
+CREATE DATABASE kubearchive WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE_PROVIDER = libc LOCALE = 'en_US.UTF-8';
+ALTER DATABASE kubearchive SET "TimeZone" TO 'UTC';
+
+CREATE USER kubearchive WITH ENCRYPTED PASSWORD 'Databas3Passw0rd';
+ALTER DATABASE kubearchive OWNER TO kubearchive;
+ALTER DATABASE kubearchive SET work_mem TO '64MB';
diff --git a/pkg/database/database.go b/pkg/database/database.go
index 332e76dd1..ae32d1006 100644
--- a/pkg/database/database.go
+++ b/pkg/database/database.go
@@ -12,6 +12,7 @@ import (
 	"github.com/kubearchive/kubearchive/pkg/database/sql"
 )
 
+var CurrentDatabaseSchemaVersion = "1"
 var RegisteredDatabases = map[string]interfaces.Database{
 	"postgresql": sql.NewPostgreSQLDatabase(),
 	"mariadb":    sql.NewMariaDBDatabase(),
diff --git a/pkg/database/fake/database_generated.go b/pkg/database/fake/database_generated.go
index 58f099605..9ca0445ea 100644
--- a/pkg/database/fake/database_generated.go
+++ b/pkg/database/fake/database_generated.go
@@ -9,6 +9,7 @@ import (
 	"errors"
 	"time"
 
+	"github.com/kubearchive/kubearchive/pkg/database"
 	"github.com/kubearchive/kubearchive/pkg/models"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/types"
@@ -86,6 +87,10 @@ func (f *fakeDatabase) TestConnection(_ map[string]string) error {
 	return f.err
 }
 
+func (f *fakeDatabase) QueryDatabaseSchemaVersion(ctx context.Context) (string, error) {
+	return database.CurrentDatabaseSchemaVersion, nil
+}
+
 func (f *fakeDatabase) queryResources(_ context.Context, kind, version, _, _ string, _ int) []*unstructured.Unstructured {
 	return f.filterResourcesByKindAndApiVersion(kind, version)
 }
diff --git a/pkg/database/interfaces/database.go b/pkg/database/interfaces/database.go
index fe91518ba..edcdb11ca 100644
--- a/pkg/database/interfaces/database.go
+++ b/pkg/database/interfaces/database.go
@@ -16,6 +16,7 @@ type DBReader interface {
 		name, continueId, continueDate string, labelFilters *models.LabelFilters, limit int) ([]string, int64, string, error)
 	QueryLogURL(ctx context.Context, kind, apiVersion, namespace, name, containerName string) (string, string, error)
 	Ping(ctx context.Context) error
+	QueryDatabaseSchemaVersion(ctx context.Context) (string, error)
 	CloseDB() error
 	Init(env map[string]string) error
 }
@@ -24,6 +25,7 @@ type DBWriter interface {
 	WriteResource(ctx context.Context, k8sObj *unstructured.Unstructured, data []byte, lastUpdated time.Time) error
 	WriteUrls(ctx context.Context, k8sObj *unstructured.Unstructured, jsonPath string, logs ...models.LogTuple) error
 	Ping(ctx context.Context) error
+	QueryDatabaseSchemaVersion(ctx context.Context) (string, error)
 	CloseDB() error
 	Init(env map[string]string) error
 }
diff --git a/pkg/database/sql/database.go b/pkg/database/sql/database.go
index 09ab8049f..db78ccb95 100644
--- a/pkg/database/sql/database.go
+++ b/pkg/database/sql/database.go
@@ -46,6 +46,13 @@ func (db *sqlDatabaseImpl) Init(env map[string]string) error {
 	return nil
 }
 
+func (db *sqlDatabaseImpl) QueryDatabaseSchemaVersion(ctx context.Context) (string, error) {
+	strQueryPerformer := newQueryPerformer[string](db.db, db.flavor)
+
+	sb := db.selector.VersionSelector()
+	return strQueryPerformer.performSingleRowQuery(ctx, sb)
+}
+
 func (db *sqlDatabaseImpl) Ping(ctx context.Context) error {
 	return db.db.PingContext(ctx)
 }
diff --git a/pkg/database/sql/facade/selector.go b/pkg/database/sql/facade/selector.go
index 858685d02..3d5d8070d 100644
--- a/pkg/database/sql/facade/selector.go
+++ b/pkg/database/sql/facade/selector.go
@@ -12,6 +12,7 @@ type DBSelector interface {
 	OwnedResourceSelector() *sqlbuilder.SelectBuilder
 	UrlFromResourceSelector() *sqlbuilder.SelectBuilder
 	UrlSelector() *sqlbuilder.SelectBuilder
+	VersionSelector() *sqlbuilder.SelectBuilder
 }
 
 // PartialDBSelectorImpl implements partially the DBSelector interface
@@ -35,3 +36,8 @@ func (PartialDBSelectorImpl) UrlSelector() *sqlbuilder.SelectBuilder {
 	sb := sqlbuilder.NewSelectBuilder()
 	return sb.Select("url", "json_path").From("log_url")
 }
+
+func (PartialDBSelectorImpl) VersionSelector() *sqlbuilder.SelectBuilder {
+	sb := sqlbuilder.NewSelectBuilder()
+	return sb.Select("version").From("schema_migrations")
+}


Reviews:
user: login: skoved
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T14:10:03Z
user: login: maruiz93
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-17T08:52:23Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T10:31:33Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T09:17:18Z
user: login: maruiz93
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-31T17:35:17Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:03:34Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:03:50Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:04:38Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:04:46Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:05:48Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:05:57Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:06:41Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:07:10Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:09:35Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:12:17Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:13:25Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:13:34Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:14:20Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T08:15:43Z
user: login: ggallen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T12:10:14Z
user: login: skoved
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-01T12:27:51Z
user: login: mafh314
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T09:51:53Z
user: login: mafh314
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T10:07:53Z
user: login: mafh314
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T10:10:53Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T07:25:39Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T07:26:43Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T07:27:31Z
user: login: ggallen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T12:14:33Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T12:27:17Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T11:50:05Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T12:00:41Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T12:10:13Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T12:19:00Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T12:25:42Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T09:36:25Z
user: login: maruiz93
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T14:39:53Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T07:19:04Z
user: login: maruiz93
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-10T08:17:54Z
user: login: skoved
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-17T12:25:47Z
user: login: ggallen
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T02:28:09Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:31:23Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:36:11Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:36:17Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:36:26Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:36:42Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:36:56Z
user: login: rh-hemartin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T08:37:02Z
user: login: ggallen
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T19:19:02Z

Review comments:
diff_hunk: @@ -0,0 +1,47 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+. Stop KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment kubearchive-sink kubearchive-api-server kubearchive-operator --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale back KubeArchive:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: skoved
body: I think it should be scale up

```suggestion
. Apply the new KubeArchive manifests and scale up KubeArchive:
```
created_at: 2025-03-14T13:55:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: skoved
body: ```suggestion
* Undo DB changes if something goes wrong
```
created_at: 2025-03-14T13:56:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: skoved
body: ```suggestion
**Note**: the implementation of database migrations do not include the execution of those
migrations. We believe that production databases should not be modified in an unattended way
so we expect the KubeArchive operator to execute the migrations.
```
created_at: 2025-03-14T14:00:02Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: skoved
body: ```suggestion
* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
```
created_at: 2025-03-14T14:01:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: skoved
body: ```suggestion
We decided to use the
link:https://github.com/golang-migrate/migrate[`migrate` tool]
due to its popularity and because it's well maintained. It automates certain
processes related with migrations, but it's a thin layer that can be easily understood
and replaced if needed.
```
created_at: 2025-03-14T14:05:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports within its integration
+folder (`integrations/database/<db-name>/migrations`. Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: skoved
body: ```suggestion
The migrations files are provided for each database KubeArchive supports in its integration
folder (`integrations/database/<db-name>/migrations`). Each migration has two files: `up` and `down`,
which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
those changes. Each file should start with a two digit number and an underscore (for example
`01_initial.up.sql` and `01_initial.down.sql`).
```
created_at: 2025-03-14T14:08:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports within its integration
+folder (`integrations/database/<db-name>/migrations`. Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
+
+== Execution
+
+To apply migrations with `migrate` the command line looks like this:
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: I suggest a more schematic view:

```suggestion
=== Apply all migrations to get the schema up-to-date
```
created_at: 2025-03-17T08:28:00Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports within its integration
+folder (`integrations/database/<db-name>/migrations`. Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
+
+== Execution
+
+To apply migrations with `migrate` the command line looks like this:
+
+[source,bash]
+----
+migrate \
+    -path migrations/ \
+    -database <driver>://<username>:<password>@<ip/dns>:<port>/<database> \
+    up
+----
+
+The command `up` runs all non-applied migrations. To rollback one version, use `down 1`:
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: Following the previous comment suggestion:

```suggestion
=== Rollback one version
```
created_at: 2025-03-17T08:28:57Z
author_association: MEMBER
diff_hunk: @@ -17,17 +17,24 @@ This document explains how to install and configure KubeArchive in your Kubernet
 
 == Prepare the Database
 
-. Create a database user `kubearchive`.
-. With a DB admin account, apply the schema provided in the [filename]`kubearchive.sql` file
-within the KubeArchive release:
+. Create a user and a password for KubeArchive to use.
+. Create a database for KubeArchive, with admin permissions granted to the user you just created.
path: docs/modules/ROOT/pages/getting-started/installation.adoc
user: login: maruiz93
body: ```suggestion
. Create a database for KubeArchive, with admin permissions granted to new user.
```
created_at: 2025-03-17T08:29:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports within its integration
+folder (`integrations/database/<db-name>/migrations`. Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
+
+== Execution
+
+To apply migrations with `migrate` the command line looks like this:
+
+[source,bash]
+----
+migrate \
+    -path migrations/ \
+    -database <driver>://<username>:<password>@<ip/dns>:<port>/<database> \
+    up
+----
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: Is there a way to know how many migrations are we applying with this command so we know how many we should downgrade in a rollback?

Also, what happens if one of the migration fails? It rollbacks to the initial state?
created_at: 2025-03-17T08:30:53Z
author_association: MEMBER
diff_hunk: @@ -33,7 +33,10 @@ while ! nc -vz localhost ${LOCAL_PORT} > /dev/null 2>&1 ; do
     sleep 0.5
 done
 echo .
-psql -h localhost -U postgres -p ${LOCAL_PORT} -f kubearchive.sql
+psql -h localhost -U postgres -p ${LOCAL_PORT} -f setup.sql
+
+export PGPASSWORD="Databas3Passw0rd"
path: integrations/database/postgresql/install.sh
user: login: maruiz93
body: We probably need the # notsecret comment in here

```suggestion
export PGPASSWORD="Databas3Passw0rd"  # notsecret
```
created_at: 2025-03-17T08:34:10Z
author_association: MEMBER
diff_hunk: @@ -77,13 +80,40 @@ func NewDatabase() (DBInterface, error) {
 		return nil, fmt.Errorf("no database registered with name '%s'", env[DbKindEnvVar])
 	}
 
+	version, err := database.QueryDatabaseSchemaVersion(context.Background())
+	if err != nil {
+		// We close because the parent caller don't close if there is any error
+		if err := database.CloseDB(); err != nil {
+			return nil, errors.New("error closing the database while retrieving the schema version")
+		}
+		return nil, errors.New("cant retrieve schema version, make sure you run the migrations")
+	}
+
+	slog.Debug("retrieved database version information", "version", version)
+
+	if version != CurrentDatabaseSchemaVersion {
+		// We close because the parent caller don't close if there is any error
+		if err := database.CloseDB(); err != nil {
+			return nil, errors.New("error closing the database after retrieving the schema version")
+		}
+		return nil, fmt.Errorf("the schema is on version '%s' but the expected is '%s'. Please run the database migrations", version, CurrentDatabaseSchemaVersion)
+	}
+
 	return database, nil
 }
 
 func (db *Database) Ping(ctx context.Context) error {
 	return db.DB.PingContext(ctx)
 }
 
+func (db *Database) QueryDatabaseSchemaVersion(ctx context.Context) (string, error) {
+	strQueryPerformer := newQueryPerformer[string](db.DB, db.Flavor)
+
+	sb := sqlbuilder.NewSelectBuilder()
+	sb.Select("version").From("schema_migrations")
+	return strQueryPerformer.performSingleRowQuery(ctx, sb)
path: pkg/database/database.go
user: login: maruiz93
body: Following the [pattern established for the database](https://kubearchive.github.io/kubearchive/main/design/database-interface.html) to have more re-utilization of the queries, we should avoid creating the queries in this file.

```suggestion
	sb := db.Selector.VersionSelector()
        return strQueryPerformer.performSingleRowQuery(ctx, sb)
```

and also add the selector in `pkg/database/facade/selector.go`:

```
type DBSelector interface {
    ...
    VersionSelector() *sqlbuilder.SelectBuilder
}

func (PartialDBSelectorImpl) VersionSelector() *sqlbuilder.SelectBuilder {
	sb := sqlbuilder.NewSelectBuilder()
	return sb.Select("version").From("schema_migrations")
}
```
created_at: 2025-03-17T08:43:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,47 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: I'd suggest adding an "Important" callout to inform the user that KubeArchive isn't backward compatible with database changes.

For example:

```suggestion
[IMPORTANT]
====
KubeArchive is not backward compatible with database changes.

Both application and schema must be upgraded at the same time.
====
```
created_at: 2025-03-17T08:51:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Grant the ability of undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we leave to the KubeArchive operator the migration executions.
+
+== Requirements
+
+* Migrations are run by the PostgreSQL user that KubeArchive uses, to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because its maintained. This tool automates certain
+processes related with migrations, but its a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports within its integration
+folder (`integrations/database/<db-name>/migrations`. Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
+
+== Execution
+
+To apply migrations with `migrate` the command line looks like this:
+
+[source,bash]
+----
+migrate \
+    -path migrations/ \
+    -database <driver>://<username>:<password>@<ip/dns>:<port>/<database> \
+    up
+----
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: > Is there a way to know how many migrations are we applying with this command so we know how many we should downgrade in a rollback?

You can see how many they get applied when you apply them:

```
$ migrate -path migrations/ -database postgresql://kubearchive:${PGPASSWORD}@localhost:${LOCAL_PORT}/kubearchive up
Handling connection for 5433
2/u fake (43.07758ms)
```

So you know you need to go back 1:

```
$ migrate -path migrations/ -database postgresql://kubearchive:${PGPASSWORD}@localhost:${LOCAL_PORT}/kubearchive down 1
Handling connection for 5433
2/d fake (27.17098ms)
```

I'm creating a section to handle failures and rollbacks.
created_at: 2025-03-17T10:31:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fail the database version is modified to the failed migration but
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: ```suggestion
When a migration fails the database version is modified to the failed migration but
```
created_at: 2025-03-18T09:13:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: I'd suggest using label filtering as we already have labels for all the kubearchive components:

```suggestion
kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
```
created_at: 2025-03-31T16:50:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: Same as before:

```suggestion
kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
```
created_at: 2025-03-31T16:51:06Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: By "dirty" do you mean with possible data inconsistencies? 
created_at: 2025-03-31T16:52:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+After you fixed the problem, you need to force the database version with
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: Suggestion to be more clear:

```suggestion
When you fix the problem, force the migration to the desired schema version with
```
created_at: 2025-03-31T16:55:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+After you fixed the problem, you need to force the database version with
+link:https://github.com/golang-migrate/migrate/blob/master/GETTING_STARTED.md#forcing-your-database-version[the following command]
+(continuing the PostgreSQL example):
+
+[source,bash]
+----
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    force <VERSION>
+----
+
+[WARNING]
+====
+If you fixed the database removing what the failed migration did and then fixed
+the migration file you need to force `<VERSION - 1>`. If you fixed by completing
+the failed migration then force `<VERSION>`. After `force` run `up` again
+to make sure that all the migrations are applied.
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: ```suggestion
The schema version for `migrate force` must be:

* `<VERSION -1>` when you rolled back the changes of `migrate` and you override the migration file.
* `<VERSION>` when you manually completed the failed `migrate` execution.

Then run `migrate up` again to make sure you apply the whole migration stack.
```
created_at: 2025-03-31T17:04:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: Active tense and  suggestion to change `operator` to a word that doesn't collide with a KubeArchive component.
```suggestion
**Note**: the implementation of database migrations does not include the execution of those
migrations. We believe that you should not modify production databases in an unattended way
so we expect the KubeArchive maintainer to execute the migrations.
```
created_at: 2025-03-31T17:07:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: Active tense:
```suggestion
* KubeArchive's PostgreSQL admin runs the migrations to avoid permission problems.
```
created_at: 2025-03-31T17:16:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates certain
+processes related with migrations, but it's a thin layer that can be easily understood
+and replaced if needed.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: I wouldn't use "easily understood" we shouldn't assume how hard a process can be for someone that isn't familiar enough with it.
```suggestion
due to its popularity and because it's well maintained. It automates part of the migration process but you can
replace it if needed.
```
created_at: 2025-03-31T17:23:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates certain
+processes related with migrations, but it's a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports in its integration
+folder (`integrations/database/<db-name>/migrations`). Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: maruiz93
body: Active tense with a more schematic description:

```suggestion
Each database engine integration provides a `migrations` folder (`integrations/database/<db-name>/migrations`). 
All migrations have two files: `up` for upgrades and `down` for rollbacks. 
Each file should start with a two digit number and an underscore (for example
`01_initial.up.sql` and `01_initial.down.sql`).
```
created_at: 2025-03-31T17:26:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,58 @@
+BEGIN;
+
+CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+BEGIN
+    NEW.updated_at = NOW();
+RETURN NEW;
+END;
+    $$;
+
+CREATE TABLE public.resource (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid UNIQUE NOT NULL,
+    api_version character varying NOT NULL,
+    kind character varying NOT NULL,
+    name character varying NOT NULL,
+    namespace character varying NOT NULL,
+    resource_version character varying,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL,
+    cluster_deleted_ts timestamp with time zone,
+    data jsonb NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE TABLE public.log_url (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid NOT NULL REFERENCES public.resource(uuid) ON DELETE CASCADE,
+    url text NOT NULL,
+    container_name text NOT NULL,
+    json_path text,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE INDEX idx_creation_timestamp_id ON public.resource
+    USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
+
+CREATE INDEX idx_json_annotations ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)) jsonb_path_ops);
path: integrations/database/postgresql/migrations/01_initial_schema.up.sql
user: login: maruiz93
body: After https://github.com/kubearchive/kubearchive/issues/903 this should be the standard index:

```suggestion
    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)) );
```
created_at: 2025-03-31T17:29:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,58 @@
+BEGIN;
+
+CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+BEGIN
+    NEW.updated_at = NOW();
+RETURN NEW;
+END;
+    $$;
+
+CREATE TABLE public.resource (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid UNIQUE NOT NULL,
+    api_version character varying NOT NULL,
+    kind character varying NOT NULL,
+    name character varying NOT NULL,
+    namespace character varying NOT NULL,
+    resource_version character varying,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL,
+    cluster_deleted_ts timestamp with time zone,
+    data jsonb NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE TABLE public.log_url (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid NOT NULL REFERENCES public.resource(uuid) ON DELETE CASCADE,
+    url text NOT NULL,
+    container_name text NOT NULL,
+    json_path text,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE INDEX idx_creation_timestamp_id ON public.resource
+    USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
+
+CREATE INDEX idx_json_annotations ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)) jsonb_path_ops);
+
+CREATE INDEX idx_json_labels ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'labels'::text)) jsonb_path_ops);
path: integrations/database/postgresql/migrations/01_initial_schema.up.sql
user: login: maruiz93
body: Same as before:

```suggestion
CREATE INDEX idx_json_labels ON public.resource
    USING gin ((((data -> 'metadata'::text) -> 'labels'::text)) );
```
created_at: 2025-03-31T17:30:08Z
author_association: MEMBER
diff_hunk: @@ -23,18 +23,21 @@ import (
 
 const defaultContainerAnnotation = "kubectl.kubernetes.io/default-container"
 
-var ResourceNotFoundError = errors.New("resource not found")
-
 type newDatabaseFunc func(*sqlx.DB) DBInterface
 type newDBCreatorFunc func(map[string]string) facade.DBCreator
 
-var RegisteredDatabases = make(map[string]newDatabaseFunc)
-var RegisteredDBCreators = make(map[string]newDBCreatorFunc)
+var (
+	ResourceNotFoundError        = errors.New("resource not found")
path: pkg/database/database.go
user: login: maruiz93
body: I guess that with the new linter config this should be renamed to:

```suggestion
	ErrResourceNotFound        = errors.New("resource not found")
```
created_at: 2025-03-31T17:31:14Z
author_association: MEMBER
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: maruiz93
body: I'm not sure why are we deleting this file, isn't this needed for the API and the sink to establish the db connection?
created_at: 2025-03-31T17:34:58Z
author_association: MEMBER
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: rh-hemartin
body: I changed the approach to use a SQL file to create the user since we need a SQL to create the DB with the correct properties. We can keep the user creation using this file if you want.
created_at: 2025-04-01T08:03:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -23,18 +23,21 @@ import (
 
 const defaultContainerAnnotation = "kubectl.kubernetes.io/default-container"
 
-var ResourceNotFoundError = errors.New("resource not found")
-
 type newDatabaseFunc func(*sqlx.DB) DBInterface
 type newDBCreatorFunc func(map[string]string) facade.DBCreator
 
-var RegisteredDatabases = make(map[string]newDatabaseFunc)
-var RegisteredDBCreators = make(map[string]newDBCreatorFunc)
+var (
+	ResourceNotFoundError        = errors.New("resource not found")
path: pkg/database/database.go
user: login: rh-hemartin
body: I will rebase soon to pick up those changes.
created_at: 2025-04-01T08:03:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,58 @@
+BEGIN;
+
+CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+BEGIN
+    NEW.updated_at = NOW();
+RETURN NEW;
+END;
+    $$;
+
+CREATE TABLE public.resource (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid UNIQUE NOT NULL,
+    api_version character varying NOT NULL,
+    kind character varying NOT NULL,
+    name character varying NOT NULL,
+    namespace character varying NOT NULL,
+    resource_version character varying,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL,
+    cluster_deleted_ts timestamp with time zone,
+    data jsonb NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE TABLE public.log_url (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid NOT NULL REFERENCES public.resource(uuid) ON DELETE CASCADE,
+    url text NOT NULL,
+    container_name text NOT NULL,
+    json_path text,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE INDEX idx_creation_timestamp_id ON public.resource
+    USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
+
+CREATE INDEX idx_json_annotations ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)) jsonb_path_ops);
path: integrations/database/postgresql/migrations/01_initial_schema.up.sql
user: login: rh-hemartin
body: Fixed, thanks
created_at: 2025-04-01T08:04:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,58 @@
+BEGIN;
+
+CREATE FUNCTION public.trigger_set_timestamp() RETURNS trigger
+    LANGUAGE plpgsql
+    AS $$
+BEGIN
+    NEW.updated_at = NOW();
+RETURN NEW;
+END;
+    $$;
+
+CREATE TABLE public.resource (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid UNIQUE NOT NULL,
+    api_version character varying NOT NULL,
+    kind character varying NOT NULL,
+    name character varying NOT NULL,
+    namespace character varying NOT NULL,
+    resource_version character varying,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL,
+    cluster_deleted_ts timestamp with time zone,
+    data jsonb NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.resource FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE TABLE public.log_url (
+    id BIGSERIAL PRIMARY KEY,
+    uuid uuid NOT NULL REFERENCES public.resource(uuid) ON DELETE CASCADE,
+    url text NOT NULL,
+    container_name text NOT NULL,
+    json_path text,
+    created_at timestamp with time zone DEFAULT now() NOT NULL,
+    updated_at timestamp with time zone DEFAULT now() NOT NULL
+);
+
+CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.log_url FOR EACH ROW EXECUTE FUNCTION public.trigger_set_timestamp();
+
+CREATE INDEX idx_creation_timestamp_id ON public.resource
+    USING btree ((((data -> 'metadata'::text) ->> 'creationTimestamp'::text)) DESC, id DESC);
+
+CREATE INDEX idx_json_annotations ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'annotations'::text)) jsonb_path_ops);
+
+CREATE INDEX idx_json_labels ON public.resource
+    USING gin ((((data -> 'metadata'::text) -> 'labels'::text)) jsonb_path_ops);
path: integrations/database/postgresql/migrations/01_initial_schema.up.sql
user: login: rh-hemartin
body: Fixed, thanks.
created_at: 2025-04-01T08:04:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Changed.
created_at: 2025-04-01T08:05:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Changed.
created_at: 2025-04-01T08:05:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+After you fixed the problem, you need to force the database version with
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Changed.
created_at: 2025-04-01T08:06:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+After you fixed the problem, you need to force the database version with
+link:https://github.com/golang-migrate/migrate/blob/master/GETTING_STARTED.md#forcing-your-database-version[the following command]
+(continuing the PostgreSQL example):
+
+[source,bash]
+----
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    force <VERSION>
+----
+
+[WARNING]
+====
+If you fixed the database removing what the failed migration did and then fixed
+the migration file you need to force `<VERSION - 1>`. If you fixed by completing
+the failed migration then force `<VERSION>`. After `force` run `up` again
+to make sure that all the migrations are applied.
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Yep, its better this way. Thanks.
created_at: 2025-04-01T08:07:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Hm... I don't like the word maintainer here. I understand what you mean, but we already have "maintainer" as someone that maintains KubeArchive's code. "user" may be better than "operator" but I think that will lead to discuss what is a user
created_at: 2025-04-01T08:09:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Changed to use the active tense. Note that this is not referring to the human, this is referring to the user account that runs the actual SQL script of the migration.
created_at: 2025-04-01T08:12:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates certain
+processes related with migrations, but it's a thin layer that can be easily understood
+and replaced if needed.
+
+The migrations files are provided for each database KubeArchive supports in its integration
+folder (`integrations/database/<db-name>/migrations`). Each migration has two files: `up` and `down`,
+which enables rollbacks. The `up` file applies changes in tables and data and the `down` files undoes
+those changes. Each file should start with a two digit number and an underscore (for example
+`01_initial.up.sql` and `01_initial.down.sql`).
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Changed.
created_at: 2025-04-01T08:13:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* Migrations are run by KubeArchive's PostgreSQL user to avoid permission problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates certain
+processes related with migrations, but it's a thin layer that can be easily understood
+and replaced if needed.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Changed.
created_at: 2025-04-01T08:13:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,56 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations do not include the execution of those
+migrations. We believe that production databases should not be modified in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: I changed the text but kept "operator"
created_at: 2025-04-01T08:14:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: I mean with data and schema inconsistencies. `migrate` does not do anything when faced with a error. This is why I wrapped everything into a single transaction, if something fails PostgreSQL won't apply anything.
created_at: 2025-04-01T08:15:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: ggallen
body: I think you meant to delete `create-user.sql`?  I think we do need this file. It's not creating a user, but a secret with the needed credentials.
created_at: 2025-04-01T12:10:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+When you fixe the problem, force the migration to the desired version with
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: mafh314
body: I think there is a typo here: `fixe`
created_at: 2025-04-02T09:51:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,13 +80,39 @@ func NewDatabase() (DBInterface, error) {
 		return nil, fmt.Errorf("no database registered with name '%s'", env[DbKindEnvVar])
 	}
 
+	version, err := database.QueryDatabaseSchemaVersion(context.Background())
+	if err != nil {
+		// We close because the parent caller don't close if there is any error
+		if err := database.CloseDB(); err != nil {
+			return nil, errors.New("error closing the database while retrieving the schema version")
+		}
+		return nil, errors.New("cant retrieve schema version, make sure you run the migrations")
path: pkg/database/database.go
user: login: mafh314
body: typo: cant -> can't or cannot or can not.
created_at: 2025-04-02T10:07:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,12 +12,6 @@ spec:
     # columns: connection-type database user address auth-options
     pg_hba:
       - hostnossl  all  all  all  reject
-  bootstrap:
path: integrations/database/postgresql/postgres.yaml
user: login: mafh314
body: if we keep the kubearchive-user, should we keep these lines too?
created_at: 2025-04-02T10:10:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: @maruiz93 
created_at: 2025-04-04T07:25:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: rh-hemartin
body: We don't need it, I changed the implementation and we don't need to extract the credentials from this secret.
created_at: 2025-04-04T07:26:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -12,12 +12,6 @@ spec:
     # columns: connection-type database user address auth-options
     pg_hba:
       - hostnossl  all  all  all  reject
-  bootstrap:
path: integrations/database/postgresql/postgres.yaml
user: login: rh-hemartin
body: In case we keep it yes, but I don't think we need to keep it.
created_at: 2025-04-04T07:27:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: ggallen
body: What change is that?
created_at: 2025-04-04T12:14:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: rh-hemartin
body: Its the change on the `postgres.yaml` on the integration's folder. I changed it so it initializes with a superuser account that we then use to create the `kubearchive` user. With this change the kubearchive account does not have superuser permissions so we don't miss on any permission.
created_at: 2025-04-04T12:27:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: maruiz93
body: ok so if I understand this correctly now the db setup for the test environment is:

1. Start the DB with a superuser
2. Use the superuser to create a kubearchive db user and make it own the kubearchive database within postgres

I get this part but my question is that `api` and `sink` will still need the `kubearchive-user` Secret to be mounted in their pods so they know the credentials to connect to the database, right?

Am I missing something here?
created_at: 2025-04-07T11:50:05Z
author_association: MEMBER
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: rh-hemartin
body: Your understanding is correct: the DB starts with a superuser, we use that superuser to create the kubearchive user and database.

The `api` and `sink` rely on the `kubearchive-database-credentials` secret to connect to the database, not the `kubearchive-user`secret.
created_at: 2025-04-07T12:00:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: maruiz93
body: yes, I've just checked that by doing the process from your branch
created_at: 2025-04-07T12:10:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: maruiz93
body: sorry for the delay, what do you think about explicitly say that there could be schema and data inconsistencies so it's more clear what to expect?

```
When a migration fails, the version of the database changes but it will be in a state with data and/or schema
inconsistencies. That sops `migrate` from applying more migrations:
```
created_at: 2025-04-07T12:17:31Z
author_association: MEMBER
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: maruiz93
body: I'm thinking that maybe we should do this differently. We have the `kubearchive-database-credentials` Secret already setup for the postgresql integration instead of patching the resource as we do with other integrations. 

In our [docs](https://kubearchive.github.io/kubearchive/main/getting-started/installation.html#_install_and_configure_kubearchive) we are stating that we expect the user to apply the release and then path the `Secret` so I think we should do that in the postgresql integration too and make the default `kubearchive-database-credentials` Secret empty.
created_at: 2025-04-07T12:25:41Z
author_association: MEMBER
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: rh-hemartin
body: I agree but I won't change the process even more in this PR, I think its already disruptive. If you want to push for that change open a ticket.
created_at: 2025-04-08T09:36:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,13 +0,0 @@
-# Copyright KubeArchive Authors
path: integrations/database/postgresql/kubearchive-user.yaml
user: login: maruiz93
body: https://github.com/kubearchive/kubearchive/issues/1005 
created_at: 2025-04-09T14:39:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,94 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment \
+    kubearchive-sink \
+    kubearchive-api-server \
+    kubearchive-operator \
+    --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails the database version is modified to the failed migration but
+left into a "dirty" state. That stops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Fixed
created_at: 2025-04-10T07:19:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails, the version of the database changes but it is left in a
+state with data and/or schema inconsistencies. That sops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: ggallen
body: Typo "sops".  I think it should be "stops".
created_at: 2025-04-20T23:24:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails, the version of the database changes but it is left in a
+state with data and/or schema inconsistencies. That sops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+When you fix the problem, force the migration to the desired version with
+link:https://github.com/golang-migrate/migrate/blob/master/GETTING_STARTED.md#forcing-your-database-version[the following command]
+(continuing the PostgreSQL example):
+
+[source,bash]
+----
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    force <VERSION>
+----
+
+[WARNING]
+====
+The schema version for `migrate force` must be:
+* `<VERSION -1>` when you rolled back the changes of `migrate` and you override the migration file.
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: ggallen
body: I this should be "overrode" to match the tense.
created_at: 2025-04-20T23:24:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: ggallen
body: I think this should be "runs" instead of "run".
created_at: 2025-04-20T23:29:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: ggallen
body: This should be "implementations".
created_at: 2025-04-20T23:39:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates part of the migration
+process but you can replace it if needed.
+
+Each database engine integration provides a `migrations` folder (`integrations/database/<db-name>/migrations`).
+All migrations have two files: `up` for upgrades and `down` for rollbacks.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: ggallen
body: I see both "rollback" and "downgrade" used.  Should we pick one?
created_at: 2025-04-20T23:40:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,17 +17,24 @@ This document explains how to install and configure KubeArchive in your Kubernet
 
 == Prepare the Database
 
-. Create a database user `kubearchive`.
-. With a DB admin account, apply the schema provided in the [filename]`kubearchive.sql` file
-within the KubeArchive release:
+. Create a user and a password for KubeArchive to use.
+. Create a database for KubeArchive, with admin permissions granted to the new user.
+  Configure its timezone as UTC and its locale as UTF-8.
path: docs/modules/ROOT/pages/getting-started/installation.adoc
user: login: ggallen
body: Why is the timezone important? And why do we say to create the database here as it looks like it's being created in `setup.sql`?
created_at: 2025-04-20T23:42:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,7 +31,6 @@ func NewWriter() (interfaces.DBWriter, error) {
 
 func newDatabase() (interfaces.Database, error) {
 	var err error
-
path: pkg/database/database.go
user: login: ggallen
body: White-space change!
created_at: 2025-04-22T02:27:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,17 +17,24 @@ This document explains how to install and configure KubeArchive in your Kubernet
 
 == Prepare the Database
 
-. Create a database user `kubearchive`.
-. With a DB admin account, apply the schema provided in the [filename]`kubearchive.sql` file
-within the KubeArchive release:
+. Create a user and a password for KubeArchive to use.
+. Create a database for KubeArchive, with admin permissions granted to the new user.
+  Configure its timezone as UTC and its locale as UTF-8.
path: docs/modules/ROOT/pages/getting-started/installation.adoc
user: login: rh-hemartin
body: The document `getting-started/installation` describes what users should do to install KubeArchive, not what developers do. So `setup.sql` is not executed by the user, `setup.sql` is replicating what a user wanting to install KubeArchive should do. 

Maybe the timezone is not really important, but normally distributed systems are set to UTC.
created_at: 2025-04-22T08:31:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails, the version of the database changes but it is left in a
+state with data and/or schema inconsistencies. That sops `migrate` from applying more migrations:
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Fixed.
created_at: 2025-04-22T08:36:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Fixed.
created_at: 2025-04-22T08:36:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
+
+== Implementation
+
+We decided to use the
+link:https://github.com/golang-migrate/migrate[`migrate` tool]
+due to its popularity and because it's well maintained. It automates part of the migration
+process but you can replace it if needed.
+
+Each database engine integration provides a `migrations` folder (`integrations/database/<db-name>/migrations`).
+All migrations have two files: `up` for upgrades and `down` for rollbacks.
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Fixed. I picked downgrade.
created_at: 2025-04-22T08:36:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,7 +31,6 @@ func NewWriter() (interfaces.DBWriter, error) {
 
 func newDatabase() (interfaces.Database, error) {
 	var err error
-
path: pkg/database/database.go
user: login: rh-hemartin
body: Reverted!
created_at: 2025-04-22T08:36:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,54 @@
+= Database Migrations
+
+This document details the database migration strategy KubeArchive uses. The main goals
+of the migrations are:
+
+* Make the upgrade experience easier
+* Undo DB changes if something goes wrong
+
+**Note**: the implementation of database migrations does not include the execution of those
+migrations. We believe that you should not modify production databases in an unattended way
+so we expect the KubeArchive operator to execute the migrations.
+
+== Requirements
+
+* The same database user that KubeArchive uses run the migrations to avoid permissions problems.
+* Migrations migrate data as well as the schema.
+* Migrations have forward and backwards implementation (`up` and `down` files).
path: docs/modules/ROOT/pages/design/database-migrations.adoc
user: login: rh-hemartin
body: Fixed
created_at: 2025-04-22T08:36:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,86 @@
+= Upgrading KubeArchive
+
+This document details the process to upgrade your KubeArchive installation.
+
+[IMPORTANT]
+====
+* KubeArchive is not backward compatible with database changes.
+  Both application and schema must be upgraded at the same time.
+* Backup your database before doing a KubeArchive upgrade.
+====
+
+. Scale down KubeArchive:
++
+[source,bash]
+----
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 0
+----
+
+. Download the KubeArchive repository and switch to the version you would like to upgrade
+to.
++
+[source,bash]
+----
+git clone https://github.com/kubearchive/kubearchive.git --branch=v1.1.0 kubearchive-v1.1.0
+----
+
+. Run the database migrations:
++
+[source,bash]
+----
+cd kubearchive-v1.1.0
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    up
+----
+
+. Apply the new KubeArchive manifests and scale up KubeArchive:
++
+[source,bash]
+----
+kubectl apply -n kubearchive \
+    -f https://github.com/kubearchive/kubearchive/releases/download/v1.1.0/kubearchive.yaml
+kubectl scale -n kubearchive deployment -l app.kubernetes.io/part-of=kubearchive --replicas 1
+----
+
+. Check pods/deployments are running correctly:
++
+[source,bash]
+----
+kubectl get -n kubearchive all
+kubectl logs -n kubearchive deployment/kubearchive-sink
+kubectl logs -n kubearchive deployment/kubearchive-api-server
+kubectl logs -n kubearchive deployment/kubearchive-operator
+----
+
+== Handling Upgrade Or Downgrade Failure
+
+When a migration fails, the version of the database changes but it is left in a
+state with data and/or schema inconsistencies. That sops `migrate` from applying more migrations:
+
+[source,text]
+----
+error: Dirty database version <VERSION>. Fix and force version.
+----
+
+When this happens inspect the state of the database to decide how to proceed.
+When you fix the problem, force the migration to the desired version with
+link:https://github.com/golang-migrate/migrate/blob/master/GETTING_STARTED.md#forcing-your-database-version[the following command]
+(continuing the PostgreSQL example):
+
+[source,bash]
+----
+migrate -verbose \
+    -path integrations/database/postgresql/migrations/ \
+    -database postgresql://<kubearchive-user>:<kubearchive-password>@<postgresql-host>:<postgresql-port>/<kubearchive-database> \
+    force <VERSION>
+----
+
+[WARNING]
+====
+The schema version for `migrate force` must be:
+* `<VERSION -1>` when you rolled back the changes of `migrate` and you override the migration file.
path: docs/modules/ROOT/pages/configuration/upgrading.adoc
user: login: rh-hemartin
body: Fixed.
created_at: 2025-04-22T08:37:02Z
author_association: CONTRIBUTOR

PR comments:
user: login: mafh314
created_at: 2025-04-02T10:16:50Z
body: Hey @rh-hemartin I have a question regarding the [README.md](https://github.com/kubearchive/kubearchive/blob/main/integrations/database/postgresql/README.md) that we have in postgresql folder. 
We have instructions there with pg_dump to recreate the kubearchive.sql file. Should we update or delete those instrucions?
author_association: CONTRIBUTOR
user: login: rh-hemartin
created_at: 2025-04-08T09:41:22Z
body: @ggallen I want your input in this PR as well, I know you already commented, but I think you did not have time yet to review the full thing. If you want I can walk you trough the changes.
author_association: CONTRIBUTOR