Repository: memgraph/helm-charts
Pull request #141: Add core dumps PVC
Author: login: gitbuda
Labels: name: breaking
Created: 2025-05-05T17:22:05Z | Merged: 2025-05-12T08:26:44Z

 - [x] Figure out the `chown` failures
- [x] Make sure we can get the core dumps at least under Linux host
- [x] Define the right naming, NOW: `/var/core/memgraph`
- [x] Figure out why CI is failing -> seems like this was the problem with setting up the permissions.
    - [x] `tee: /proc/sys/kernel/yama/ptrace_scope: No such file or directory` (this failed locally, but that's not the failure under the CI) -> check if the file is there first
    - [x] Test AKS (Azure) and other apply other code review suggestions
    - [x] Apply `find . -path ./lost+found -prune -o -exec chown -hR username:group {} +` -> doesn't work revert back to `|| true`.
- [x] Add user group under HA.
- [x] Add AKS init script. 
- [x] Seems like the user:group of the lost+found repo got changed... -> debug -> the problem was that find returns `.` and then `chown -R` again finds `lost+found` directory.

### Important Considerations:
* Security: Core dumps can contain sensitive data from your application's memory. Ensure that the storage location for core dumps is properly secured and access is restricted.   
* Resource Usage: Core dumps can be large, especially for memory-intensive applications. Ensure that your storage solution has sufficient capacity and consider limiting the size of core dumps.
* Node Configuration: Modifying the host node's core_pattern often requires privileged access and can affect all processes running on that node, not just your Kubernetes pods. Be cautious when making such changes.
* Container Filesystem Permissions: Ensure that the application within the container has the necessary permissions to write core dump files to the desired location.

### Example
#### For Application-Specific Core Dumps
If your application runs as a specific user and needs to write its own core dumps:
```bash
mkdir -p /var/core/memgraph
chown memgraph:memgraph /var/core/memgraph
```
#### Configuring Core Dump Settings
You'll also want to configure the system's core dump settings:
```bash
ulimit -c unlimited
echo '/var/crash/core.%e.%p.%h.%t' | tee  /proc/sys/kernel/core_pattern
echo "0" | tee /proc/sys/kernel/yama/ptrace_scope
```

### Research Pages
* https://wiki.archlinux.org/title/Core_dump


Diff:
diff --git a/README.md b/README.md
index 991c02e..a3136b7 100644
--- a/README.md
+++ b/README.md
@@ -94,5 +94,11 @@ Contributions are welcome! If you have any improvements, bug fixes, or new chart
   </a>
 </p>
 
+## Debugging Memgraph Pods
+
+Find more details under [Debugging Running
+Pods](https://memgraph.com/docs/database-management/debugging#debugging-running-pods)
+documentation section.
+
 ## License
 This repository is licensed under the [Apache 2.0 License](https://github.com/memgraph/helm-charts/blob/main/LICENSE).
diff --git a/charts/memgraph-high-availability/templates/coordinators.yaml b/charts/memgraph-high-availability/templates/coordinators.yaml
index 404cb25..19b1c91 100644
--- a/charts/memgraph-high-availability/templates/coordinators.yaml
+++ b/charts/memgraph-high-availability/templates/coordinators.yaml
@@ -82,7 +82,18 @@ spec:
         - name: memgraph-coordinator-{{ $coordinator.id }}-log-storage
           mountPath: /var/log/memgraph
         command: [ "/bin/sh","-c" ]
-        args: [ "chown -R memgraph:memgraph /var/log/memgraph /var/lib/memgraph || true" ]
+        # The permissions have to be explicitly adjusted because under some k8s
+        # environments permissions set under
+        # https://github.com/memgraph/memgraph/blob/master/release/debian/postinst
+        # get overwritten. Sometimes, PVC are created using new partitions ->
+        # lost+found directory should not change its permissions so it has to
+        # be excluded.
+        args:
+          - >
+            cd /var/log/memgraph;
+            find . -path ./lost+found -prune -o -exec chown {{ $.Values.memgraphUserGroupId }} {} +;
+            cd /var/lib/memgraph;
+            find . -path ./lost+found -prune -o -exec chown {{ $.Values.memgraphUserGroupId }} {} +;
         securityContext:
           readOnlyRootFilesystem: true
           runAsUser: 0 # Run as root
diff --git a/charts/memgraph-high-availability/templates/data.yaml b/charts/memgraph-high-availability/templates/data.yaml
index 230cc08..016fc81 100644
--- a/charts/memgraph-high-availability/templates/data.yaml
+++ b/charts/memgraph-high-availability/templates/data.yaml
@@ -92,7 +92,18 @@ spec:
         - name: memgraph-data-{{ $data.id }}-log-storage
           mountPath: /var/log/memgraph
         command: [ "/bin/sh","-c" ]
-        args: [ "chown -R memgraph:memgraph /var/log/memgraph /var/lib/memgraph || true" ]
+        # The permissions have to be explicitly adjusted because under some k8s
+        # environments permissions set under
+        # https://github.com/memgraph/memgraph/blob/master/release/debian/postinst
+        # get overwritten. Sometimes, PVC are created using new partitions ->
+        # lost+found directory should not change its permissions so it has to
+        # be excluded.
+        args:
+          - >
+            cd /var/log/memgraph;
+            find . -path ./lost+found -prune -o -exec chown {{ $.Values.memgraphUserGroupId }} {} +;
+            cd /var/lib/memgraph;
+            find . -path ./lost+found -prune -o -exec chown {{ $.Values.memgraphUserGroupId }} {} +;
         securityContext:
           readOnlyRootFilesystem: true
           runAsUser: 0 # Run as root
diff --git a/charts/memgraph-high-availability/values.yaml b/charts/memgraph-high-availability/values.yaml
index d4fddf3..d1b6482 100644
--- a/charts/memgraph-high-availability/values.yaml
+++ b/charts/memgraph-high-availability/values.yaml
@@ -75,6 +75,12 @@ sysctlInitContainer:
   enabled: true
   maxMapCount: 262144
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
+
 secrets:
   enabled: false
   name: memgraph-secrets
diff --git a/charts/memgraph/templates/statefulset.yaml b/charts/memgraph/templates/statefulset.yaml
index 43ba060..aa88321 100644
--- a/charts/memgraph/templates/statefulset.yaml
+++ b/charts/memgraph/templates/statefulset.yaml
@@ -26,6 +26,7 @@ spec:
       annotations:
         {{- toYaml . | nindent 4 }}
       {{- end }}
+
     spec:
       {{- if .Values.serviceAccount.create }}
       serviceAccount: {{ include "memgraph.serviceAccountName" . }}
@@ -48,18 +49,34 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
+          # The permissions have to be explicitly adjusted because under some
+          # k8s environments permissions set under
+          # https://github.com/memgraph/memgraph/blob/master/release/debian/postinst
+          # get overwritten. Sometimes, PVC are created using new partitions ->
+          # lost+found directory should not change its permissions so it has to
+          # be excluded.
           args:
             - >
               {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
+              cd /var/lib/memgraph;
+              find . -path ./lost+found -prune -o -exec chown {{ .Values.memgraphUserGroupId }} {} +;
               {{- end }}
               {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
+              cd /var/log/memgraph;
+              find . -path ./lost+found -prune -o -exec chown {{ .Values.memgraphUserGroupId }} {} +;
               {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              cd {{ .Values.persistentVolumeClaim.userMountPath }};
+              find . -path ./lost+found -prune -o -exec chown {{ .Values.memgraphUserGroupId }} {} +;
+              {{- end }}
+              {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+              cd {{ .Values.persistentVolumeClaim.coreDumpsMountPath }};
+              find . -path ./lost+found -prune -o -exec chown {{ .Values.memgraphUserGroupId }} {} +;
               {{- end }}
           securityContext:
             readOnlyRootFilesystem: true
@@ -67,6 +84,7 @@ spec:
             capabilities:
               drop: ["ALL"]
               add: ["CHOWN"]
+
         {{- if .Values.sysctlInitContainer.enabled }}
         - name: init-sysctl
           image: busybox
@@ -74,7 +92,21 @@ spec:
           securityContext:
             privileged: true
             runAsUser: 0
-          {{- end }}
+        {{- end }}
+
+        {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+        - name: init-core-dumps
+          image: busybox
+          command: ['/bin/sh', '-c']
+          args:
+            - >
+              echo '{{ .Values.persistentVolumeClaim.coreDumpsMountPath }}/core.%e.%p.%t.%s' | tee /proc/sys/kernel/core_pattern;
+              if [ -d /proc/sys/kernel/yama ]; then echo '0' | tee /proc/sys/kernel/yama/ptrace_scope; fi
+          securityContext:
+            privileged: true
+            runAsUser: 0
+        {{- end }}
+
       terminationGracePeriodSeconds: {{ .Values.container.terminationGracePeriodSeconds }}
       securityContext:
       {{- if .Values.useImagePullSecrets }}
@@ -101,6 +133,11 @@ spec:
           persistentVolumeClaim:
             claimName: {{ include "memgraph.fullname" . }}-user-storage
       {{- end }}
+      {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+        - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+          persistentVolumeClaim:
+            claimName: {{ include "memgraph.fullname" . }}-core-dumps-storage
+      {{- end }}
       {{- range .Values.customQueryModules }}
         - name: {{ .volume | quote }}
           configMap:
@@ -164,6 +201,10 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-log-storage
               mountPath: /var/log/memgraph
           {{- end }}
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           {{- if .Values.persistentVolumeClaim.createUserClaim }}
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
@@ -210,6 +251,7 @@ spec:
         volumeName: {{ .Values.persistentVolumeClaim.storageVolumeName }}
         {{- end }}
   {{- end }}
+
   {{- if .Values.persistentVolumeClaim.createLogStorage }}
     - metadata:
         name: {{ include "memgraph.fullname" . }}-log-storage
@@ -224,6 +266,20 @@ spec:
             storage: {{ .Values.persistentVolumeClaim.logStorageSize }}
   {{- end }}
 
+  {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+    - metadata:
+        name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+      spec:
+        accessModes:
+        - "ReadWriteOnce"
+        {{- if .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        storageClassName: {{ .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        {{- end }}
+        resources:
+          requests:
+            storage: {{ .Values.persistentVolumeClaim.coreDumpsStorageSize }}
+  {{- end }}
+
   {{- if .Values.persistentVolumeClaim.createUserClaim }}
     - metadata:
         name: {{ include "memgraph.fullname" . }}-user-storage
diff --git a/charts/memgraph/values.yaml b/charts/memgraph/values.yaml
index c34a9d6..31a8d97 100644
--- a/charts/memgraph/values.yaml
+++ b/charts/memgraph/values.yaml
@@ -79,6 +79,12 @@ persistentVolumeClaim:
   userStorageAccessMode: "ReadWriteOnce"
   userMountPath:
 
+  ## Create a Persistant Volume Claim for core dumps.
+  createCoreDumpsClaim: false
+  coreDumpsStorageClassName:
+  coreDumpsStorageSize: 10Gi
+  coreDumpsMountPath: /var/core/memgraph
+
 # Default Storage Class for data and logs, defaults are for Minikube, make sure to change it for production deployments
 # Examples provisioner: Minikube(k8s.io/minikube-hostpath) AWS (ebs.csi.aws.com), GCP (pd.csi.storage.gke.io), Azure (disk.csi.azure.com)
 # Examples storageType: Minikube(hostPath) AWS (gp2), GCP (pd-standard), Azure (StandardSSD_LRS)
@@ -98,6 +104,12 @@ memgraphConfig:
 # the pod. For further information, check the `resources` section in this file about setting pod memory and cpu limits.
 - "--also-log-to-stderr=true"
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
+
 secrets:
   enabled: false
   name: memgraph-secrets
diff --git a/scripts/aks.bash b/scripts/aks.bash
new file mode 100755
index 0000000..5d90195
--- /dev/null
+++ b/scripts/aks.bash
@@ -0,0 +1,34 @@
+#!/bin/bash -e
+
+RESOURCE_GROUP="${RESOURCE_GROUP:-TestingResourceGroup}"
+LOCATION="${LOCATION:-northeurope}"
+CLUSTER_NAME="${CLUSTER_NAME:-memgraph-standalone}"
+CLUSTER_SIZE="${CLUSTER_SIZE:-1}"
+NODE_VM_SIZE="${NODE_VM_SIZE:-Standard_A2_v2}"
+
+# NOTE: Assumes installed az and being logged in
+# https://learn.microsoft.com/en-us/cli/azure/install-azure-cli.
+
+create_cluster() {
+  az group create --name $RESOURCE_GROUP --location $LOCATION
+  az aks create --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME \
+    --node-count $CLUSTER_SIZE --node-vm-size $NODE_VM_SIZE --generate-ssh-keys
+  az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME
+}
+
+delete_cluster() {
+  az group delete --name $RESOURCE_GROUP --yes
+}
+
+case $1 in
+  create_cluster)
+    create_cluster
+  ;;
+  delete_cluster)
+    delete_cluster
+  ;;
+  *)
+    echo "$0 create_cluster|delete_cluster"
+    exit 1
+  ;;
+esac


Reviews:
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T15:25:04Z
user: login: as51340
body: I think creating directories with mkdir -p shouldn't be needed.
Currently this cannot be started on AKS because you are missing chown on log and lib volumes. 
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T07:25:17Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T08:13:02Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T08:13:52Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T08:26:12Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T09:34:36Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T19:38:36Z
user: login: antejavor
body: Just for context it looks like it works in minikube: 

```
memgraph@memgraph-db-0:/var/core$ ls -la
total 12
drwxr-xr-x 3 root     root     4096 May  8 21:36 .
drwxr-xr-x 1 root     root     4096 May  8 21:36 ..
drwxrwxrwx 2 memgraph memgraph 4096 May  8 21:35 memgraph
```

```
➜  helm-charts git:(f28b969) kubectl logs memgraph-db-0 -c init-core-dumps
/var/core/memgraph/core.%e.%p.%t.%s
0

```
```
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m31s  default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Normal   Scheduled         4m29s  default-scheduler  Successfully assigned default/memgraph-db-0 to minikube
  Normal   Pulling           4m29s  kubelet            Pulling image "memgraph/memgraph:3.2.0"
  Normal   Pulled            3m39s  kubelet            Successfully pulled image "memgraph/memgraph:3.2.0" in 49.653s (49.653s including waiting)
  Normal   Created           3m39s  kubelet            Created container init-volume-mounts
  Normal   Started           3m39s  kubelet            Started container init-volume-mounts
  Normal   Pulling           3m39s  kubelet            Pulling image "busybox"
  Normal   Pulled            3m37s  kubelet            Successfully pulled image "busybox" in 1.71s (1.71s including waiting)
  Normal   Created           3m37s  kubelet            Created container init-sysctl
  Normal   Started           3m37s  kubelet            Started container init-sysctl
  Normal   Pulling           3m37s  kubelet            Pulling image "busybox"
  Normal   Pulled            3m35s  kubelet            Successfully pulled image "busybox" in 1.624s (1.624s including waiting)
  Normal   Created           3m35s  kubelet            Created container init-core-dumps
  Normal   Started           3m35s  kubelet            Started container init-core-dumps
  Normal   Pulled            3m35s  kubelet            Container image "memgraph/memgraph:3.2.0" already present on machine
  Normal   Created           3m34s  kubelet            Created container memgraph
  Normal   Started           3m34s  kubelet            Started container memgraph
  ```
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T22:09:46Z
user: login: antejavor
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T22:12:38Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T17:30:46Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-10T12:56:51Z
user: login: as51340
body: Looks good
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-12T06:46:18Z
user: login: as51340
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T06:50:12Z
user: login: gitbuda
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T08:25:49Z

Review comments:
diff_hunk: @@ -61,20 +65,35 @@ spec:
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
               chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
               {{- end }}
+              {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+              mkdir -p {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.coreDumpsMountPath }} || true;
+              {{- end }}
           securityContext:
-            readOnlyRootFilesystem: true
+            readOnlyRootFilesystem: true # TODO(gitbuda): The above chown commands fail...
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: put this back
created_at: 2025-05-06T15:25:03Z
author_association: MEMBER
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
+              chown -R {{ .Values.memgraphUserGroupId }} {{ .Values.persistentVolumeClaim.userMountPath }};
path: charts/memgraph/templates/statefulset.yaml
user: login: as51340
body: I think || true needs to stay because of a directory called lost+found created by K8s. Changing permissions on lost+found is not allowed and hence init container will fail although it changed permissions successfully on memgraph directories
created_at: 2025-05-08T07:19:08Z
author_association: COLLABORATOR
diff_hunk: @@ -98,6 +104,12 @@ memgraphConfig:
 # the pod. For further information, check the `resources` section in this file about setting pod memory and cpu limits.
 - "--also-log-to-stderr=true"
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
path: charts/memgraph/values.yaml
user: login: as51340
body: will this stay like this? If yes then I will change it also for HA chart
created_at: 2025-05-08T07:19:34Z
author_association: COLLABORATOR
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
+              chown -R {{ .Values.memgraphUserGroupId }} {{ .Values.persistentVolumeClaim.userMountPath }};
path: charts/memgraph/templates/statefulset.yaml
user: login: as51340
body: you are missing chown if log storage exists and if createStorageClaim true
created_at: 2025-05-08T07:23:25Z
author_association: COLLABORATOR
diff_hunk: @@ -98,6 +104,12 @@ memgraphConfig:
 # the pod. For further information, check the `resources` section in this file about setting pod memory and cpu limits.
 - "--also-log-to-stderr=true"
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
path: charts/memgraph/values.yaml
user: login: gitbuda
body: yes, I was thinking about that, was waiting on the comment here, because these ids are a hack even on the Memgraph side... -> will do
created_at: 2025-05-08T08:13:02Z
author_association: MEMBER
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
+              chown -R {{ .Values.memgraphUserGroupId }} {{ .Values.persistentVolumeClaim.userMountPath }};
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: aha that's why there was the ||... -> but that's also super misleading, will experiment what's the best there

what do you mean when you say "log storage exists" (there is already a folder there)? putting back the chown is fine I just didn't understand why are they needed...
created_at: 2025-05-08T08:13:52Z
author_association: MEMBER
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
+              chown -R {{ .Values.memgraphUserGroupId }} {{ .Values.persistentVolumeClaim.userMountPath }};
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: Instead of doing `|| true` probably a better way is `find . -path ./lost+found -prune -o -exec chown -hR username:group {} +`

It's there because AKS is obviously mounting new partitions for each claim/volume (not 100% but it doesn't matter).
created_at: 2025-05-08T08:26:12Z
author_association: MEMBER
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: if `mountPath` automatically creates this folder -> remove it from here
created_at: 2025-05-08T09:34:36Z
author_association: MEMBER
diff_hunk: @@ -48,33 +49,54 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
-            - >
+            - > # Document or change the || true.
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: Document or replace `|| true`
created_at: 2025-05-08T19:38:36Z
author_association: MEMBER
diff_hunk: @@ -98,6 +104,12 @@ memgraphConfig:
 # the pod. For further information, check the `resources` section in this file about setting pod memory and cpu limits.
 - "--also-log-to-stderr=true"
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
path: charts/memgraph/values.yaml
user: login: antejavor
body: The `chown` currently works because we are using a memgraph image for init containers also, so there is user group and user `memgraph:memgraph`, but we should probably change this to  `busybox` and hardcode. 
created_at: 2025-05-08T22:06:07Z
author_association: COLLABORATOR
diff_hunk: @@ -79,6 +79,12 @@ persistentVolumeClaim:
   userStorageAccessMode: "ReadWriteOnce"
   userMountPath:
 
+  ## Create a Persistant Volume Claim for core dumps.
+  createCoreDumpsClaim: true
path: charts/memgraph/values.yaml
user: login: antejavor
body: Don't forget to change the default,  since I guess you probably do not want this PVC by default. 
created_at: 2025-05-08T22:12:38Z
author_association: COLLABORATOR
diff_hunk: @@ -48,33 +49,50 @@ spec:
             - name: {{ include "memgraph.fullname" . }}-user-storage
               mountPath: {{ .Values.persistentVolumeClaim.userMountPath }}
           {{- end }}
-
+          {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+            - name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+              mountPath: {{ .Values.persistentVolumeClaim.coreDumpsMountPath }}
+          {{- end }}
           command: ["/bin/sh", "-c"]
           args:
             - >
-              {{- if .Values.persistentVolumeClaim.createStorageClaim }}
-              chown -R memgraph:memgraph /var/lib/memgraph || true;
-              {{- end }}
-              {{- if .Values.persistentVolumeClaim.createLogStorage }}
-              chown -R memgraph:memgraph /var/log/memgraph || true;
-              {{- end }}
               {{- if .Values.persistentVolumeClaim.createUserClaim }}
-              chown -R memgraph:memgraph {{ .Values.persistentVolumeClaim.userMountPath }} || true;
+              mkdir -p {{ .Values.persistentVolumeClaim.createUserClaim }};
+              chown -R {{ .Values.memgraphUserGroupId }} {{ .Values.persistentVolumeClaim.userMountPath }};
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: find doesn't work -> reverted back to || true
created_at: 2025-05-09T17:30:46Z
author_association: MEMBER
diff_hunk: @@ -98,6 +104,12 @@ memgraphConfig:
 # the pod. For further information, check the `resources` section in this file about setting pod memory and cpu limits.
 - "--also-log-to-stderr=true"
 
+# The explicit user and group setup is required because at the init container
+# time, there is not yet a user created. This seems fine because under both
+# Memgraph and Mage images we actually hard-code the user and group id. The
+# config is used to chown user storage and core dumps claims' month paths.
+memgraphUserGroupId: "100:101"
path: charts/memgraph/values.yaml
user: login: gitbuda
body: changing the container itself should be another PR to avoid additional complexities here
created_at: 2025-05-10T12:56:51Z
author_association: MEMBER
diff_hunk: @@ -224,6 +266,20 @@ spec:
             storage: {{ .Values.persistentVolumeClaim.logStorageSize }}
   {{- end }}
 
+  {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+    - metadata:
+        name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+      spec:
+        accessModes:
+        - "ReadWriteOnce"
+        {{- if .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        storageClassName: {{ .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        {{- end }}
path: charts/memgraph/templates/statefulset.yaml
user: login: as51340
body: I think you don't need to check if storage class name exists. If the value is empty, the default storage class will be used. Not that important but I use it like that in ha chart
created_at: 2025-05-12T06:46:13Z
author_association: COLLABORATOR
diff_hunk: @@ -224,6 +266,20 @@ spec:
             storage: {{ .Values.persistentVolumeClaim.logStorageSize }}
   {{- end }}
 
+  {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+    - metadata:
+        name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+      spec:
+        accessModes:
+        - "ReadWriteOnce"
+        {{- if .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        storageClassName: {{ .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        {{- end }}
path: charts/memgraph/templates/statefulset.yaml
user: login: as51340
body: btw tested on AKS, seems good
created_at: 2025-05-12T06:50:12Z
author_association: COLLABORATOR
diff_hunk: @@ -224,6 +266,20 @@ spec:
             storage: {{ .Values.persistentVolumeClaim.logStorageSize }}
   {{- end }}
 
+  {{- if .Values.persistentVolumeClaim.createCoreDumpsClaim }}
+    - metadata:
+        name: {{ include "memgraph.fullname" . }}-core-dumps-storage
+      spec:
+        accessModes:
+        - "ReadWriteOnce"
+        {{- if .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        storageClassName: {{ .Values.persistentVolumeClaim.coreDumpsStorageClassName }}
+        {{- end }}
path: charts/memgraph/templates/statefulset.yaml
user: login: gitbuda
body: Aha, yes, I'll leave this as is, but in the v3.2.1 PR, I will apply and test that.
created_at: 2025-05-12T08:25:49Z
author_association: MEMBER

PR comments:
user: login: gitbuda
created_at: 2025-05-06T20:12:17Z
body: @antejavor and @as51340, I consider the PR done, please review 🙏 
author_association: MEMBER
user: login: as51340
created_at: 2025-05-07T06:35:08Z
body: A good "smoke test" is to run it on AKS. Did you try to do that?
author_association: COLLABORATOR
user: login: gitbuda
created_at: 2025-05-07T08:21:27Z
body: I still have to fix the CI run (I'm not sure why it's failing). Once I do that, can I ask you to run on AKS? Because I'll probably spend a day figuring out how to run there...
author_association: MEMBER
user: login: as51340
created_at: 2025-05-07T09:41:34Z
body: CI fails probably because of linter. I will check linter and AKS then np. 
author_association: COLLABORATOR
user: login: gitbuda
created_at: 2025-05-08T08:17:32Z
body: > I think creating directories with mkdir -p shouldn't be needed. Currently this cannot be started on AKS because you are missing chown on log and lib volumes.

The reasoning with mkdir is that user could use that folder for something completely random, some additional files or maybe adding new query modules during runtime

I don't understand why is `chown` needed? Under Kind everything is already setup, as I think it should be, but yea, we'll put it back if it's really the best solution.
author_association: MEMBER
user: login: gitbuda
created_at: 2025-05-08T08:27:04Z
body: @as51340 do you know why CI test lint run is failing? I couldn't reproduce that locally...
author_association: MEMBER
user: login: as51340
created_at: 2025-05-12T06:48:29Z
body: In general: this change will be breaking in a sense that users will have to do `uninstall` and then `install` to do a chart upgrade. Upgrading with the `helm upgrade` won't be possible I think because of groupUserId change
author_association: COLLABORATOR
user: login: gitbuda
created_at: 2025-05-12T08:26:09Z
body: Btw, I've also tested under GKE -> also works 🎉 
author_association: MEMBER