GitLab/Cheat Sheet

More or less loose collection of GitLab commands that might be helpful (or were helpful at some point)
== GitLab ==
=== Restarts ===
Reboot GitLab instance:
cookbook sre.hosts.reboot-single gitlab1004.wikimedia.org
Restart GitLab service:
gitlab-ctl restart
Reload GitLab config:
gitlab-ctl reconfigure
Restart (git) SSH daemon:
systemctl restart ssh-gitlab
Check GitLab status:
gitlab-ctl status
Get GitLab logs (see also https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server):
sudo gitlab-ctl tail
=== Backup and Restore ===
Trigger a full backup:
systemctl start full-backup
Trigger a config backup:
systemctl start config-backup
Get logs of backup:
journalctl -u full-backup
Sync backup to replica (assuming gitlab1003 is the replica):
systemctl start rsync-data-backup-gitlab1003.wikimedia.org.service
systemctl start rsync-config-backup-gitlab1003.wikimedia.org.service
Trigger a restore on replica:
systemctl start backup-restore
Check status of restore:
systemctl status backup-restore.service
Check logs of restore:
journalctl -u backup-restore.service
=== Postgres ===
Connect to the embedded Postgres database:
sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
== gitlab-runner ==
=== Restarts ===
Gracefully reboot all GitLab Runner:
cookbook sre.gitlab.reboot-runner --reason "reason" --alias gitlab-runner reboot
Restart gitlab-runner service:
systemctl restart gitlab-runner
=== Depool/pause a runner ===
Run the depool command on the host (/usr/local/bin/depool)
=== Pool/unpause a runner ===
Run the pool command on the host (/usr/local/bin/pool)
=== Register/Unregister: ===
Unregister Runner (note, this can also be done by setting profile::gitlab::runner::ensure to absent and run puppet):
gitlab-runner unregister --name `hostname -f`
Unregister Trusted Runner (different because running as non-root:
su -s /bin/bash gitlab-runner -c "gitlab-runner unregister --name `hostname -f`"
Register Runner (handled by puppet):
run-puppet-agent
=== Clean docker cache: ===
Cleanup old docker volumes in /var/lib/docker:
/usr/share/gitlab-runner/clear-docker-cache
=== Registering to a replica: ===
If you need to test something on a replica using a runner, it can be done easily
# Pause the runner on the admin interface of the primary (https://gitlab.wikimedia.org/admin/runners)
# Disable puppet: 
disable-puppet "Re-registering with replica temporarily"
# Stop the runner
systemctl stop gitlab-runner
# Change the host the runner registers with in `config.toml`, `config-template.toml`, `managed.toml`, and `registration.toml`
sudo sed -i -e 's/gitlab.wikimedia.org/gitlab-replica-b.wikimedia.org/' /etc/gitlab-runner/*.toml
# Start the runner:
systemctl start gitlab-runner
After this, you can re-enable puppet to register the runner with the primary again
=== Run commands against all WMCS Runners (cloudcumin) ===
cloudcumin host can be used to run commands against all WMCS Runners. Login to cloudcumin:
sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'command'
For example show all runner system_ids:
sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'cat /etc/gitlab-runner/.runner_system_id'
==== Depool/pause all WMCS runners ====
Login to cloudcumin and run:
sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'depool'
==== Pool/unpause all WMCS runners ====
Login to cloudcumin and run:
sudo cumin 'O{project:gitlab-runners name:^runner.*}' 'pool'