PullMirrorsOverdueQueueTooLarge in staging

https://ops.gitlab.net/gitlab-com/runbooks/blob/master/troubleshooting/large-pull-mirror-queue.md states to check the mirror dashboard, but there is no data for staging.

UpdateAllMirrorsWorker had 116 entries when I checked.

Since this is staging, those can't be customer jobs - so I cleared the set.

states = ProjectImportState.where(project_id: projects).order(:last_update_started_at).map(&:last_error) Returned the following:

```
=> ["Import timed out. Import took longer than 54000 seconds", "Import timed out. Import took longer than 54000 seconds", nil, nil, "error: cannot lock ref 'refs/remotes/upstream/old_version': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ithkuil/ithkuil.git/./refs/remotes/upstream/old_version.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/site': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ithkuil/ithkuil.git/./refs/remotes/upstream/site.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/site-python2': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ithkuil/ithkuil.git/./refs/remotes/upstream/site-python2.lock': No space left on device\n", "error: cannot lock ref 'refs/remotes/upstream/master': Unable to create '/var/opt/gitlab/git-data-file03/repositories/RoliSoft/dotfiles.git/./refs/remotes/upstream/master.lock': No space left on device\n", nil, "error: cannot lock ref 'refs/remotes/upstream/form2': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/form2.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/fs2851': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/fs2851.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/large-screens': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/large-screens.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/lessnew': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/lessnew.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/localdraft': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/localdraft.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/mail': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/mail.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/mail_headers': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/mail_headers.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/master': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/master.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/mediarefactor': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/mediarefactor.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/old-stable': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/old-stable.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/pagetools': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/pagetools.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/php7comptib': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/php7comptib.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/phpunit-fix': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/phpunit-fix.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/retrytests': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/retrytests.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/scrutinizer-patch-1': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/scrutinizer-patch-1.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/stable': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/stable.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/tpl_action_get2': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/tpl_action_get2.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/travis': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/travis.lock': No space left on device\nerror: cannot lock ref 'refs/remotes/upstream/versionfixtool': Unable to create '/var/opt/gitlab/git-data-file03/repositories/ViktorBodrogi/dokuwiki.git/./refs/remotes/upstream/versionfixtool.lock': No space left on device\n", "error: cannot lock ref 'refs/remotes/upstream/master': Unable to create '/var/opt/gitlab/git-data-file03/repositories/boost/boostdep.git/./refs/remotes/upstream/master.lock': No space left on device\n", nil, nil, nil, nil, nil, nil, nil, nil, "no repository for such path", nil, "Import timed out. Import took longer than 54000 seconds", nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, "Import timed out. Import took longer than 54000 seconds", nil, "fatal: could not read Username for 'https://github.com': terminal prompts disabled\n", nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, "no repository for such path", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "no repository for such path", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass", "undefined method `commit_id' for nil:NilClass"]
```

This seems to be indicating a full disk on file-03 - but there are only 01 and 02 file nodes in staging.

I've cleared the set as specified in the runbook: https://ops.gitlab.net/gitlab-com/runbooks/blob/master/troubleshooting/large-pull-mirror-queue.md - the alert has still not cleared.

Story points: 2.0