Create three new gitaly storage shard nodes file-{90..92}-stor-gprd for storing new projects

# `Production` Change

### Change Summary

Create three new gitaly storage shard nodes `file-{90..92}-stor-gprd` for storing new projects.  Why?  Increase capacity for new project repository storage.

More details: https://gitlab.com/gitlab-com/gl-infra/capacity-planning/-/issues/507

- [ ] [Detailed steps for the change](#detailed-steps-for-the-change)
- [ ] [Build the new VM instance](#build-the-new-vm-instance)
- [ ] [Ensure the creation of the storage directory](#ensure-the-creation-of-the-storage-directory)
- [ ] [Configure the GitLab application so that it is aware of the new node](#configure-the-GitLab-application-so-that-it-is-aware-of-the-new-node)
- [ ] [Add the new Gitaly node to all our Kubernetes container configuration](#add-the-new-gitaly-node-to-all-our-kubernetes-container-configuration)
- [ ] [Test the new node](#test-the-new-node)
- [ ] [Enable the new node in Gitlab](#enable-the-new-node-in-gitlab)


### Change Details

1. **Services Impacted**  - ~"Service::Gitaly" 
1. **Change Technician**  - @nnelson
1. **Change Reviewer**    - @cmcfarland 
1. **Time tracking**      - `2 hours`
1. **Downtime Component** - `zero downtime`

## Meta

- [x] Replace all occurrences of "`XX`" with the new gitaly shard node number when executing commands.
- [x] Set the title of this production change issue to: Create new gitaly storage shard node `file-90-stor-gprd` for storing new projects


### Detailed steps for the change

The following are the detailed steps for the change.


### Build the new VM instance

- **pre-conditions for execution of the step**
  - [x] [Create a new MR](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/new).
    * The commit should increment the `"node_count" -> "default" -> "multizone-stor"` variable setting by the number of new gitlay shards that are being added [around line `533` of the file `environments/gprd/variables.tf`](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/blob/master/environments/gprd/variables.tf#L533)
    * Here is [an example title and description to use for this MR](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/1605).
    - [x] Using the new value of the `multizone-stor` field, change the MR title to: Increment multi-zone storage nodes by [Number of new gitaly shards] to [the new total]
  - [x] Link: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/5150
  - [x] Have the MR reviewed by a colleague.
- **execution commands for the step**
  - [x] Optionally, check quotas before applying the terraform changes. You can check with:
  ```bash
  gcloud --project='gitlab-production' compute regions describe us-east1 --format=json | jq -c '.quotas[] | select(.limit > 0) | select(.usage / .limit > 0.5) | { metric, limit, usage }'
  ```
  - [x] Merge the MR.
  - [x] Click the `apply gprd` pipeline stage `play` button.
- **post-execution validation for the step**
  - [x] Examine the `gprd apply` pipeline stage output and confirm the absence of relevant errors.
- **rollback of the step**
  - [ ] Revert the MR.


### Ensure the creation of the storage directory

Once the gitaly node is created, it will take a few minutes for chef to run on the system, so it may not be immediately available.

- **pre-conditions for execution of the step**
  - [x] Make sure `chef-client` runs without any errors.
  ```bash
  export node='file-90-stor-gprd.c.gitlab-production.internal'
  bundle exec knife ssh "fqdn:$node" "sudo grep 'Chef Client finished' /var/log/syslog | tail -n 1"
  ```
- **execution commands for the step**
  - [ ] **If** chef does not converge after 10 minutes or so, then invoke it manually. If chef refuses to run, then something is wrong, and this procedure should be rolled-back.
  ```bash
  bundle exec knife ssh "fqdn:$node" "sudo chef-client"
  ```
  - [x] Confirm storage directory `/var/opt/gitlab/git-data/repositories` exists on the file system of the new node.
  ```bash
  bundle exec knife ssh "fqdn:$node" "sudo df -hT /var/opt/gitlab/git-data/repositories && sudo ls -la /var/opt/gitlab/git-data/ && sudo ls -la /var/opt/gitlab/git-data/repositories | head"
  ```
- **post-execution validation for the step**
  - [x] Confirm that the gitaly service is running
  ```bash
  bundle exec knife ssh "fqdn:$node" "sudo gitlab-ctl status gitaly"
  ```
  - [x] Confirm that there are no relevant errors in the logs.
  ```bash
  bundle exec knife ssh "fqdn:$node" "sudo grep -i 'error' /var/log/gitlab/gitaly/current | tail"
  ```
- **rollback of the step**
  - No rollback procedure for this step is necessary.
  - This step only confirms and verifies steps taken so far.


### Configure the GitLab application so that it is aware of the new node

Configure the GitLab application to include the new node. Note: The GitLab application will consider the new node to be disabled by default.

- **pre-conditions for execution of the step**
  - [x] Create [a new MR in the `chef-repo` project](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/new).
    * Here is [an example title and description to use for this MR](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/1242).
    * The commit should consist of the following changes:
    - [x] Update the `override_attributes.omnibus-gitlab.gitaly.storage` list items of file [`roles/gprd-base-stor-gitaly-common.json`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base-stor-gitaly-common.json#L410), add item(s) similar to:
    ```json
                {
                  "name": "nfs-file90",
                  "path": "/var/opt/gitlab/git-data/repositories"
                },
    ```
    - [x] Update the `default_attributes.omnibus-gitlab.gitlab_rb.git_data_dirs` map entry of file [`roles/gprd-base.json`](https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-base.json#L494), add an entry similar to:
    ```json
                "nfs-file90": {
                  "path": "/var/opt/gitlab/git-data-file90",
                  "gitaly_address": "tcp://file-90-stor-gprd.c.gitlab-production.internal:9999"
                },
    ```
  - [x] Link: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/3017
  - [x] Have the MR reviewed by a colleague.
- **execution commands for the step**
  - [x] Notify the Engineer On-call about the planned change.
  - [x] Create a silence for `GitalyServiceGoserverTrafficAbsentSingleNode` alert, which will get raised if new Gitaly server(s) do not receive enough traffic for 30 minutes. [Reference of alert raised in the past](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16269#note_1100938329).
  - [x] Merge the MR.
  - [x] Examine the pipeline stage output for `apply_to_prod` job on ops.gitlab.net pipeline to verify that change was applied successfully and there were no errors.
- **post-execution validation for the step**
  - [x] Verify chef role to check for the change:
  ```bash
  $ bundle exec knife role show gprd-base-stor-gitaly-common | grep -A1 'nfs-file90'
          name: nfs-file90
          path: /var/opt/gitlab/git-data/repositories
  ```
  - [x] Wait 30-35 minutes for the nodes to converge naturally. In the normal circumstances, chef-client periodically runs every 30 (plus upto 5) minutes. Verify by checking node status (ignore patroni/postgres servers in the list):
  ```bash
  bundle exec knife status "roles:gprd-base-stor-gitaly-common" --run-list
  ```
  - [ ] Optionally, in case you are running out of patience and thinking explicit run, force `chef-client` to run on the relevant nodes (It will take excruciatingly long time though, so better to wait for natural convergence):
  ```bash
  bundle exec knife ssh -C 3 "roles:gprd-base-stor-gitaly-common" "sudo chef-client"
  ```
  
- **rollback of the step**
  - [ ] Revert the MR.
  - [ ] Check the `apply_to_prod` ops.gitlab.net pipeline to see if the change successfully applied.
  - [ ] Re-run the commands in the post-execution validation for the step


### Add the new Gitaly node to all our Kubernetes container configuration

- **pre-conditions for execution of the step**
  - [x] Create [a new MR in the `gl-infra/k8s-workloads/gitlab-com` project](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl).
    * Here is [an example title and description to use for this MR](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1779/diffs).
    - [x] In the MR you want to update the file `releases/gitlab/values/${environment}.yaml.gotmpl` and add the new node to the `global.gitaly.external` yaml list
        Typically the data looks like
    ```
    - hostname: gitaly-01-sv-pre.c.gitlab-pre.internal
      name: default
      port: "9999"
      tlsEnabled: false
    ```
  - [x] Link: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/2599
  - [x] Have the MR reviewed by a colleague in **Delivery**
- **execution commands for the step**
  - [x] Notify the Engineer On-call about the planned change and seek approval, to ensure that no other deployment (From #announcements) is ongoing at the time.
  - [x] Merge the MR.
  - [x] Examine the pipeline stage output to verify that there were no errors.

- **rollback of the step**
  - [ ] Revert the MR.
  - [ ] Re-run the execution step for a roll-back.


### Test the new node

Confirm that the new storage node is operational.

- **pre-conditions for execution of the step**
  - [ ] Export your `gitlab.com` user auth token as an environment variable in your shell session.
  ```bash
  export GITLAB_COM_API_PRIVATE_TOKEN='CHANGEME'
  ```
  - [ ] Also export your `gitlab.com` admin user auth token as an environment variable in your shell session.
  ```bash
  export GITLAB_GPRD_ADMIN_API_PRIVATE_TOKEN='CHANGEME'
  ```
- **execution commands for the step**
  - [x] Create a new project:
  ```bash
  export project_name='nfs-file90-test'
  rm -f "/tmp/project-${project_name}.json"
  curl --silent --show-error --request POST "https://gitlab.com/api/v4/projects?name=${project_name}&default_branch=main" --header "Private-Token: ${GITLAB_COM_API_PRIVATE_TOKEN}" > "/tmp/project-${project_name}.json"
  export project_id=$(cat "/tmp/project-${project_name}.json" | jq -r '.id')
  export ssh_url_to_repo=$(cat "/tmp/project-${project_name}.json" | jq -r '.ssh_url_to_repo')
  ```
  - [x] Clone the project.
  ```bash
  git clone "${ssh_url_to_repo}" "/tmp/${project_name}"
  ```
  - [x] Add, commit, and push a `README` file to the project repository.
  ```bash
  echo "# ${project_name}" > "/tmp/${project_name}/README.md"
  pushd "/tmp/${project_name}" && git add "/tmp/${project_name}/README.md" && git commit -am "Add README" && git push origin main && popd
  ```
  - [x] Use the API to [move it to a new storage server](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/gitaly/storage-rebalancing.md#manual-method):
  ```bash
  export destination_storage_name='nfs-file90'
  export move_id=$(curl --silent --show-error --request POST "https://gitlab.com/api/v4/projects/${project_id}/repository_storage_moves" --data "{\"destination_storage_name\": \"${destination_storage_name}\"}" --header "Private-Token: ${GITLAB_GPRD_ADMIN_API_PRIVATE_TOKEN}" --header 'Content-Type: application/json' | jq -r '.id')
  ```
  - [x] Optionally poll the api to monitor the state of the move:
  ```bash
  curl --silent --show-error "https://gitlab.com/api/v4/projects/${project_id}/repository_storage_moves/${move_id}" --header "Private-Token: ${GITLAB_GPRD_ADMIN_API_PRIVATE_TOKEN}" | jq -r '.state'
  ```
  - [x] Optionally confirm the new location:
  ```bash
  curl --silent --show-error "https://gitlab.com/api/v4/projects/${project_id}" --header "Private-Token: ${GITLAB_GPRD_ADMIN_API_PRIVATE_TOKEN}" | jq -r '.repository_storage'
  ```
  - [x] Once the project has finished being moved to the new shard, proceed to add, commit, and push an update to the `README`:
  ```bash
  echo -e "\n\ntest" >> "/tmp/${project_name}/README.md"
  pushd "/tmp/${project_name}" && git add "/tmp/${project_name}/README.md" && git commit -am "Update README to test nfs-file90" && git push origin main && popd
  ```
  - [x] Verify that the changes were persisted as expected:
  ```bash
  rm -rf "/tmp/${project_name}"
  git clone "${ssh_url_to_repo}" "/tmp/${project_name}"
  grep 'test' "/tmp/${project_name}/README.md"
  ```

### Enable the new node in Gitlab

Enabling new nodes in the GitLab admin console requires using an admin account to change where new projects are stored. In [`Admin Area`](https://gitlab.com/admin) > [`Settings`](https://gitlab.com/admin/application_settings/general) > [`Repository`](https://gitlab.com/admin/application_settings/repository) > `Repository storage` > `Expand`, you will see a list of storage nodes. The ones that are checked are the ones that will receive new projects. For more information see [gitlab docs](https://docs.gitlab.com/ee/administration/repository_storage_paths.html#choose-where-new-repositories-will-be-stored).

- **execution commands for the step**
  - [x] Open a private browser window or tab and navigate to: https://gitlab.com/admin/application_settings/repository
  - [x] Click the `Expand` button next to `Repository storage`.
  - [x] Click the `Save changes` button. (I know you didn't do any changes, just trust the process and click the button)
  - [x] Click play on the [Production gitaly-shard-weights-assigner job](https://ops.gitlab.net/gitlab-com/gl-infra/gitaly-shard-weights-assigner/-/pipeline_schedules) to assign a weight.

- **post-execution validation for the step**
  - [x] Take a count of how many projects are being created on the new shard:
  ```bash
  export node='file-90-stor-gprd.c.gitlab-production.internal'
  bundle exec knife ssh "fqdn:$node" "sudo find /var/opt/gitlab/git-data/repositories/@hashed -mindepth 2 -maxdepth 3 -name *.git | wc -l"
  ```
  - [x] Observe that this number goes up over time.

- **post-execution validation for the step**
  - [ ] Take a count of how many projects are being created on the old shard:
  ```bash
  export node='file-YY-stor-gprd.c.gitlab-production.internal'
  bundle exec knife ssh "fqdn:$node" "sudo find /var/opt/gitlab/git-data/repositories/@hashed -mindepth 2 -maxdepth 3 -name *.git | wc -l"
  ```
  - [ ] Observe that this number never goes up over time.  (Either goes down or does not change.)
  - [x] Delete silence created for `GitalyServiceGoserverTrafficAbsentSingleNode` alert in steps [above](#configure-the-gitlab-application-so-that-it-is-aware-of-the-new-node).

Story points: 2.0