Change:
"d ${cfg.statePath} 0750 ${cfg.user} ${cfg.group} -" "d ${cfg.statePath}/builds 0750 ${cfg.user} ${cfg.group} -" "d ${cfg.statePath}/config 0750 ${cfg.user} ${cfg.group} -" <del> "D ${cfg.statePath}/config/initializers 0750 ${cfg.user} ${cfg.group} -" </del> <ins> "d ${cfg.statePath}/config/initializers 0750 ${cfg.user} ${cfg.group} -" </ins> "d ${cfg.statePath}/db 0750 ${cfg.user} ${cfg.group} -" "d ${cfg.statePath}/log 0750 ${cfg.user} ${cfg.group} -" "d ${cfg.statePath}/repositories 2770 ${cfg.user} ${cfg.group} -"

Discussion:
Describe the bug When running a gitlab backup with gitlab-rake according to the documentation, it fails with the following log output: To Reproduce Steps to reproduce the behavior: the gitlab module ???, causing /var/gitlab/state/config/initializers to disappear -u gitlab -H gitlab-rake gitlab:backup:create RAILSENV=production Expected behavior A backup is created in /var/gitlab/state/backup/ Screenshots Additional context Metadata [mil nix run -c nix-info -m system: host os: multi-user?: sandbox: version: channels(root): channels(mil): nixpkgs: Maintainer information:
Maintainers:
talyz said on IRC they couldn't reproduce the problem. I tried to reproduce it in a VM and got this: !
I could not reproduce it in a VM either, so it's probably state corruption or something, I'll try to find out what caused it.
any success in reproducing? If you manage to trigger the segfault again, and then a backtrace might be helpful.
We could also add creating a backup to the gitlab vm test.
I think the segfault was related to the VM running out of memory ^^'
It seems like the state directory was somehow poisoned on the upgrade path from 12.1.x to 12.3.x, when creating a fresh state the backups work just fine.
Does this ring a bell?
Maybe there's something left over in the directory that should be cleaned up? If you still have the broken state available, could you try removing only the directory and see if that fixes the problem?
I backed up the database and state directory before cleaning it, but I was unable to restore it until now.
Okay, I was about to close this because resetting the state was easier than debugging it further, but now it occured again on a fresh state. What I remember I did in between: Enter my GitLab trial license Disable user sign-up Promoted my user to admin
Oh, that's interesting. The only thing in that list that I haven't done on my instance is adding a trial licence, so I'll try that and see what happens.
No I am currently trying to reproduce it in a VM and adding a trial license doesn't trigger it.

When comparing a broken installation with a fresh one, I saw that all files in /var/gitlab/state/config/initializers/ were missing, except extra-, which is empty. Copying the files from the fresh install fixes the broken one. What could cause them to go missing and how can we ensure that these file stay consistent?
On this machine, the folder is empty after each nixos-rebuild switch.
There seems to be both a tmpfiles rule and a gitlab trying to empty As systemd-tmpfiles is run on every nixos-rebuild switch, it might clean the directory after gitlab is restarted. We should only let one thing take care of doing that - maybe the gitlab service, as it also copies in the smtp initializer. The tmpfiles rule could be reduced to only create the directory, but not empty it.
Aah, I didn't notice the in the tmpfiles rule when adding the clean-up. That should definitely be fixed. I'll do that right away. Great detective work, and