Repository: mariadb-operator/mariadb-operator

Issue:
[Bug] 25.8.3 PhysicalBackup ib_logfile0 was not found
I've running latest operator and created this cr:
```
apiVersion: k8s.mariadb.com/v1alpha1
kind: PhysicalBackup
metadata:
  name: mariadb
  namespace: dev
spec:
  mariaDbRef:
    name: mariadb
  schedule:
    cron: "0 0 * * *"
    suspend: false
    immediate: true
  compression: bzip2
  maxRetention: 336h
  storage:
    s3:
      bucket: ...
      prefix: mariadb
      endpoint: ...
      region:  us-east-1
      accessKeyIdSecretKeyRef:
        name: mariadb-backup-credentials
        key: accessKey
      secretAccessKeySecretKeyRef:
        name: mariadb-backup-credentials
        key: secretKey
      tls:
        enabled: true
```
Backup pod started, log is here:
```
💾 Writing target file: /backup/0-backup-target.txt
💾 Taking backup: /backup/physicalbackup-20250820115352.xb.bz2
[00] 2025-08-20 11:54:08 Connecting to MariaDB server host: mariadb-0.mariadb-internal.dev.svc, user: root, password: set, port: 3306, socket: /run/mysqld/mysqld.sock
[00] 2025-08-20 11:54:08 Using server version 11.4.5-MariaDB-ubu2404-log
mariadb-backup based on MariaDB server 11.4.5-MariaDB debian-linux-gnu (x86_64)
[00] 2025-08-20 11:54:08 uses posix_fadvise().
[00] 2025-08-20 11:54:08 cd to /var/lib/mysql/
[00] 2025-08-20 11:54:08 open files limit requested 0, set to 1048576
[00] 2025-08-20 11:54:08 mariabackup: using the following InnoDB configuration:
[00] 2025-08-20 11:54:08 innodb_data_home_dir =
[00] 2025-08-20 11:54:08 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2025-08-20 11:54:08 innodb_log_group_home_dir = ./
[00] 2025-08-20 11:54:08 InnoDB: Using liburing
2025-08-20 11:54:08 0 [Note] InnoDB: Number of transaction pools: 1
2025-08-20 11:54:08 0 [ERROR] InnoDB: File ./ib_logfile0 was not found
[00] 2025-08-20 11:54:08 Error: cannot read redo log header
```
I see that backup mounts the same pvc as  mariadb use. Here, the ib_logfile0 exists.
```
mysql@mariadb-0:/$ ls /var/lib/mysql/ib_logfile0
/var/lib/mysql/ib_logfile0
```



Hints from issue comments:
@DrummyFloyd: i've got the same issue =/@hedgieinsocks: My hunch is that `mariadb-backup` (or at least some of its subcommands) can run only on `localhost` because they do some things with files in the data directory, not only running sql commands like `mysql-dump`.   

And in your log, the host is not `localhost`.

```
[00] 2025-08-20 11:54:08 Connecting to MariaDB server host: mariadb-0.mariadb-internal.dev.svc, user: root, password: set, port: 3306, socket: /run/mysqld/mysqld.sock
```

In other words, it might be that one should not be able to perform a successful `mariadb-backup` when supplying a remote host via `--host`.  

I have not checked the code since I'm not going to use this feature. This is just my theory, so pls take it with a grain of salt. I hope I didn't mislead you.@kenlasko: I'm having the same issue trying to do a `PhysicalBackup` to S3 using pretty much identical config to @pasztorl. Running a Galera cluster, in case that might be relevant.@hedgieinsocks: > My hunch is that mariadb-backup (or at least some of its subcommands) can run only on localhost because they do some things with files in the data directory, not only running sql commands like mysql-dump.

Per https://jira.mariadb.org/browse/MDEV-15075 using a dns host via `--host` should not cause issues since we mount the same pvc. So my guess from above appears to be unrelated. @BabyDino: Not sure if relevant, but my primary node crashes when using PhysicalBackup:

```
2025-09-03  7:00:29 0 [Note] WSREP: Running: 'wsrep_sst_mariabackup --role 'joiner' --address '10.42.3.56:4444' --datadir '/var/lib/mysql/' --parent 1 --progress 0'
/usr//bin/wsrep_sst_common: line 1837: cd: /var/lib/mysql: Permission denied
2025-09-03  7:00:29 0 [ERROR] WSREP: Failed to read 'ready <addr>' (got '(null)') from: wsrep_sst_mariabackup --role 'joiner' --address '10.42.3.56:4444' --datadir '/var/lib/mysql/' --parent 1 --progress 0
2025-09-03  7:00:29 0 [ERROR] WSREP: Process completed with error: wsrep_sst_mariabackup --role 'joiner' --address '10.42.3.56:4444' --datadir '/var/lib/mysql/' --parent 1 --progress 0: 1 (Operation not permitted)
2025-09-03  7:00:29 2 [ERROR] WSREP: Failed to prepare for 'mariabackup' SST. Unrecoverable.
2025-09-03  7:00:29 2 [ERROR] WSREP: SST request callback failed. This is unrecoverable, restart required.
```

If it mounts the same PVC, this could be the issue?@mmontes11: The issue has been investigated and resolved with the help of @kenlasko, thank you very much:
https://mariadb-community.slack.com/archives/C02R1CJ0LD6/p1757086585185059

This has been addressed by: https://github.com/mariadb-operator/mariadb-operator/pull/1400

For now, the only way to use `PhysicalBackups` with Galera is by setting `galera.config.reuseStorageVolume=false`. The setting is immutable, so unfortunately it cannot be changed after creation. @pasztorl: Thanks! what is the feature of the reuseStorageVolume? what is the use case of turning it on or off? if this is turned on PhysicalBackups will not work in the future, or there will be another change for that?@BabyDino: > Thanks! what is the feature of the reuseStorageVolume? what is the use case of turning it on or off? if this is turned on PhysicalBackups will not work in the future, or there will be another change for that?

From the description:

> ReuseStorageVolume indicates that storage volume used by MariaDB should be reused to store the Galera configuration files. It defaults to false, which implies that a dedicated volume for the Galera configuration files is provisioned.

But in my case, it already defaults to false. I have not defined it in my values. I am using Hetzner hcloud storage, could that be relevant?@mmontes11: Hey @BabyDino ! Thanks for sharing the docs. Your case looks more like a permission issue, not strictly related to the one described here. 

> /usr//bin/wsrep_sst_common: line 1837: cd: /var/lib/mysql: Permission denied

By default the container runs as `mysql` (user 999 group 999) which has write permissions in `/var/lib/mysql`. The CSIDriver relies on the [fsGroupPolicy functionality ](https://kubernetes-csi.github.io/docs/support-fsgroup.html) for changing the permissions i.e. the kubelet will recursively change the permissions to 999 in the volume before attaching in to the Pod. It requires a CSI driver with `fsGroupPolicy=File` or `fsGroupPolicy=ReadWriteOnceWithFSType`. Saying this, because I'm not sure what Hetzner hcloud uses by default.

Mind opening a different issue to handle this?@pasztorl: @BabyDino  I understand, but what is the practical benefit of it being stored separately?@kenlasko: I guess the reason is that some users want a separation between config and data. 

Solution patch:
diff --git a/pkg/builder/batch_container_builder.go b/pkg/builder/batch_container_builder.go
index 183c4c99..09fa8453 100644
--- a/pkg/builder/batch_container_builder.go
+++ b/pkg/builder/batch_container_builder.go
@@ -97,10 +97,7 @@ func jobPhysicalBackupVolumes(storageVolume mariadbv1alpha1.StorageVolumeSource,
 			},
 		},
 	})
-	volumeMounts = append(volumeMounts, corev1.VolumeMount{
-		Name:      StorageVolume,
-		MountPath: MariadbStorageMountPath,
-	})
+	volumeMounts = append(volumeMounts, mariadbStorageVolumeMount(mariadb))
 
 	return volumes, volumeMounts
 }
diff --git a/pkg/builder/container_builder.go b/pkg/builder/container_builder.go
index bd61984b..7905907e 100644
--- a/pkg/builder/container_builder.go
+++ b/pkg/builder/container_builder.go
@@ -469,6 +469,20 @@ func mariadbEnv(mariadb *mariadbv1alpha1.MariaDB) []corev1.EnvVar {
 	return env
 }
 
+func mariadbStorageVolumeMount(mariadb *mariadbv1alpha1.MariaDB) corev1.VolumeMount {
+	galera := ptr.Deref(mariadb.Spec.Galera, mariadbv1alpha1.Galera{})
+	reuseStorageVolume := ptr.Deref(galera.Config.ReuseStorageVolume, false)
+
+	storageVolumeMount := corev1.VolumeMount{
+		Name:      StorageVolume,
+		MountPath: MariadbStorageMountPath,
+	}
+	if mariadb.IsGaleraEnabled() && reuseStorageVolume {
+		storageVolumeMount.SubPath = StorageVolume
+	}
+	return storageVolumeMount
+}
+
 func mariadbVolumeMounts(mariadb *mariadbv1alpha1.MariaDB, opts ...mariadbPodOpt) []corev1.VolumeMount {
 	mariadbOpts := newMariadbPodOpts(opts...)
 	volumeMounts := []corev1.VolumeMount{
@@ -483,17 +497,7 @@ func mariadbVolumeMounts(mariadb *mariadbv1alpha1.MariaDB, opts ...mariadbPodOpt
 		volumeMounts = append(volumeMounts, tlsVolumeMounts...)
 	}
 
-	galera := ptr.Deref(mariadb.Spec.Galera, mariadbv1alpha1.Galera{})
-	reuseStorageVolume := ptr.Deref(galera.Config.ReuseStorageVolume, false)
-
-	storageVolumeMount := corev1.VolumeMount{
-		Name:      StorageVolume,
-		MountPath: MariadbStorageMountPath,
-	}
-	if mariadb.IsGaleraEnabled() && reuseStorageVolume {
-		storageVolumeMount.SubPath = StorageVolume
-	}
-	volumeMounts = append(volumeMounts, storageVolumeMount)
+	volumeMounts = append(volumeMounts, mariadbStorageVolumeMount(mariadb))
 
 	if mariadb.Replication().Enabled && ptr.Deref(mariadb.Replication().ProbesEnabled, false) {
 		volumeMounts = append(volumeMounts, corev1.VolumeMount{
@@ -511,6 +515,9 @@ func mariadbVolumeMounts(mariadb *mariadbv1alpha1.MariaDB, opts ...mariadbPodOpt
 		galeraConfigVolumeMount := corev1.VolumeMount{
 			MountPath: galeraresources.GaleraConfigMountPath,
 		}
+		galera := ptr.Deref(mariadb.Spec.Galera, mariadbv1alpha1.Galera{})
+		reuseStorageVolume := ptr.Deref(galera.Config.ReuseStorageVolume, false)
+
 		if reuseStorageVolume {
 			galeraConfigVolumeMount.Name = StorageVolume
 			galeraConfigVolumeMount.SubPath = galeraresources.GaleraConfigVolume



Test patch:
diff --git a/pkg/builder/container_builder_test.go b/pkg/builder/container_builder_test.go
index e5e265b1..dc4b1b04 100644
--- a/pkg/builder/container_builder_test.go
+++ b/pkg/builder/container_builder_test.go
@@ -1931,6 +1931,61 @@ func TestMaxscaleContainers(t *testing.T) {
 	}
 }
 
+func TestMariadbStorageVolumeMount(t *testing.T) {
+	tests := []struct {
+		name        string
+		mariadb     *mariadbv1alpha1.MariaDB
+		wantSubPath string
+	}{
+		{
+			name:        "no galera configured",
+			mariadb:     &mariadbv1alpha1.MariaDB{Spec: mariadbv1alpha1.MariaDBSpec{}},
+			wantSubPath: "",
+		},
+		{
+			name: "galera enabled reuse disabled",
+			mariadb: &mariadbv1alpha1.MariaDB{
+				Spec: mariadbv1alpha1.MariaDBSpec{
+					Galera: &mariadbv1alpha1.Galera{
+						Enabled: true,
+						GaleraSpec: mariadbv1alpha1.GaleraSpec{
+							Config: mariadbv1alpha1.GaleraConfig{
+								ReuseStorageVolume: ptr.To(false),
+							},
+						},
+					},
+				},
+			},
+			wantSubPath: "",
+		},
+		{
+			name: "galera enabled reuse enabled",
+			mariadb: &mariadbv1alpha1.MariaDB{
+				Spec: mariadbv1alpha1.MariaDBSpec{
+					Galera: &mariadbv1alpha1.Galera{
+						Enabled: true,
+						GaleraSpec: mariadbv1alpha1.GaleraSpec{
+							Config: mariadbv1alpha1.GaleraConfig{
+								ReuseStorageVolume: ptr.To(true),
+							},
+						},
+					},
+				},
+			},
+			wantSubPath: StorageVolume,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			vm := mariadbStorageVolumeMount(tt.mariadb)
+			if vm.SubPath != tt.wantSubPath {
+				t.Fatalf("unexpected SubPath: want=%q got=%q", tt.wantSubPath, vm.SubPath)
+			}
+		})
+	}
+}
+
 func defaultEnv(overrides []corev1.EnvVar) []corev1.EnvVar {
 	mysqlTcpPort := corev1.EnvVar{
 		Name:  "MYSQL_TCP_PORT",