giseldo/neodataset huggingface dataset:
A "5% disk space left" alert gets fired, but resolves almost immediately

A "5% disk space left" alert gets fired, and I get paged through PagerDuty, but then when I go to take a look at the disks on the system, everything appears normal.

```
nelsnelson@sidekiq-export-04-sv-gprd.c.gitlab-production.internal:~$ df -h
Filesystem                                                                                       Size  Used Avail Use% Mounted on
udev                                                                                              26G     0   26G   0% /dev
tmpfs                                                                                            5.2G  550M  4.6G  11% /run
/dev/sda1                                                                                         20G   11G  9.1G  54% /
tmpfs                                                                                             26G  4.0K   26G   1% /dev/shm
tmpfs                                                                                            5.0M     0  5.0M   0% /run/lock
tmpfs                                                                                             26G     0   26G   0% /sys/fs/cgroup
/dev/sdb                                                                                          50G  151M   49G   1% /var/log
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/artifacts     20T   11T  9.4T  52% /var/opt/gitlab/gitlab-rails/shared/artifacts
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-ci/builds                  20T   11T  9.4T  52% /var/opt/gitlab/gitlab-ci/builds
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/lfs-objects   20T   11T  9.4T  52% /var/opt/gitlab/gitlab-rails/shared/lfs-objects
pages-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/pages         16T  5.0T   11T  32% /var/opt/gitlab/gitlab-rails/shared/pages
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/uploads              20T   11T  9.4T  52% /var/opt/gitlab/gitlab-rails/uploads
tmpfs                                                                                            5.2G     0  5.2G   0% /run/user/1017
```

I suppose this is because the disk capacity dip is so brief, and I did not run the `df` check quickly enough.

![Screen_Shot_2019-12-08_at_10.28.35_AM](/uploads/eccbfa2bcd3905705dc14990017f9a0a/Screen_Shot_2019-12-08_at_10.28.35_AM.png)

Story points: 1.0
Failed to connect to 10.226.1.4 port 9091: Operation timed out

It seems that the deadman's switch for a few hosts may not be operational or reachable on their normal port 9091.

> Can anyone help understand what is 10.226.1.4?
> It's hardcoded in gitlab-restore as a destination for the final signal about successful restoration  https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/blob/master/common.sh#L45 – looks like it's not available anymore
```* Failed to connect to 10.226.1.4 port 9091: Operation timed out```
> So, the verification for staging backups is always failing https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/-/jobs/779052.
> Any recent changes with that address and should it be changed?
> 10.226.1.3 is also unavailable. And 10.250.8.9

-- https://gitlab.slack.com/archives/CB3LSMEJV/p1575757987020800

Story points: 1.0
Weird wobbling of web/haproxy latency+error ratios

Noticed this happening today.  The onset of the wobbling seems to correspond with the natural increase of traffic as Europe and Americas comes online.

![Screen_Shot_2019-12-06_at_10.16.46_AM](/uploads/8c2977c15a540300e6247dc48f656de5/Screen_Shot_2019-12-06_at_10.16.46_AM.png)

![Screen_Shot_2019-12-06_at_10.14.32_AM](/uploads/4a9a3cd8ae8de9fcac7d0c02ef35af51/Screen_Shot_2019-12-06_at_10.14.32_AM.png)

![Screen_Shot_2019-12-06_at_10.13.36_AM](/uploads/3bb1a6f192f62ddf90056bbc35ae479b/Screen_Shot_2019-12-06_at_10.13.36_AM.png)

![Screen_Shot_2019-12-06_at_10.13.35_AM](/uploads/09976c0a5c513bf1934226b922f9ba14/Screen_Shot_2019-12-06_at_10.13.35_AM.png)

Story points: 1.0
Upgrade to terraform GCP provider 3.x

The most recent major version upgrade of the Google Cloud Terraform provider has been released, and includes [several changes](https://github.com/terraform-providers/terraform-provider-google/blob/master/CHANGELOG.md) we will want/need in the near future. Note that this impacts how we manage GCP projects, however (see #8122)

Note that this will require a fair amount of review and testing to ensure we don't have any issues in any modules and/or when rolling out deployments across environments. This issue should be updated according to the [upgrade guide](https://www.terraform.io/docs/providers/google/guides/version_3_upgrade.html) or split into further tasks as required.

/cc @gitlab-com/gl-infra

Story points: 5.0
Automate chef-server install

Follow-up / breakout from #8028 - while the terraform code is in place for an instance in us-central, the bootstrap needs to be improved. Our current chef server was built manually, and has little to no automation for the installation and configuration of the chef server itself. There is a [chef-server cookbook](https://github.com/chef-cookbooks/chef-server) that was added to a [new role](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/chef-gitlab-net.json) based on the prior one.

With this configuration, however, we do not have any ability to test independently of deploying changes to the role onto the chef server; this issue is to track the creation of a container build/test/deployment pipeline (if possible), or a "standard" wrapper cookbook around `chef-server`, and the corresponding test-kitchen/chefspec/inspec config to validate, otherwise.

Story points: 2.0
Investigate application start failures when all known read-only replicas are not responsive.

## Summary

In the Incident Review meeting earlier today (05 December), corrective action item 13 in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8528 was skipped in favor of a more in-depth discussion.

> We need to investigate why losing just 2 patroni node affected us

The meeting was nearly over, so I stated that a follow-up meeting should be scheduled to discuss the concern. Most of the individuals in the call assembled for shortly after for a separate call, and wed ended up discussing the matter then. It became evident quickly that more information needs to be uncovered before a fruitful conversation can take place.

## Definition of Done

- [x] Outline the technical conditions under which the failure occurred.
  - [x] Provide steps to reproduce the behavior.
- [ ] Document this scenario in the appropriate section of our runbook.
- [x] @ansdval will determine a priority level and bring it to the attention of the development teams in the Performance & Availability meeting using `~infradev` and ~gitlab.com label combination.

Story points: 2.0
Use custom instance type for Patroni nodes

We currently use `n1-highmem-96` to get 600GB memory PostgreSQL server instances. This leads to very under-utilized CPU. This is currently wasting 20% of our GCP CPU quota. Not to mention the cost of reserving 1000 cores we don't use.

We can easily cut this down by using a custom node type.

![image](/uploads/648dbc4c6d5168258331866c1f164784/image.png)

Based on this "worst case" CPU utilization, we peak at about 30 cores in use. We can safely cut the allocated cores by 50% to 48 CPUs.

Proposal: Switch allocation to `custom-48-614400`.

Story points: 2.0
gitlab-restore: implement cleanup procedure to avoid "out of quota" events

It's quite common now the cases when gitlab-restore project reaches some quotas because some instances are stalled. If, during restoration, some error occurs, in some cases it doesn't lead to hard failure and auto-cleanup.

We need some job that would clean up instances periodically, based on certain mask for instance name and rules like "If the backup verification started 3 days ago and hasn't finished yet, and if the instance is not protected from deletion, it's time to consider backup verification as failed, send all the signals about it, and destroy the instance".

To discuss: how should it be organized? Cronjob or CI/CD pipelines with special task? Anyhow else?

Story points: 4.0
The "View in AlertManager" link in a PagerDuty incident is not alerts.gprd.gitlab.net, but instead ops.gitlab.net/gitlab-com/runbooks

I'd expect to go to https://alerts.gprd.gitlab.net/#/alerts or something.

But instead, for example, the "View in GitLab Alertmanager" link in https://gitlab.pagerduty.com/incidents/P5VDQCW goes to https://ops.gitlab.net/gitlab-com/runbooks/blob/master/troubleshooting/chef.md.  This is not what I would expect.

Story points: 1.0
Nodes with incomplete Chef runs

There seem to be a lot of nodes with missing chef-client metric data in gprd/gstg/etc.

https://prometheus.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=up%7Bjob%3D%22node%22%7D%20unless%20chef_client_error&g0.tab=1

Some of them have chef-client disabled. Some never complete the first client run.

Story points: 2.0
Onboard datawarehouse databases

It is needed to onboard the datawarehouse databases :

- license
- version
- customers

For this we need to execute the following steps : 
- [ ] Check hardware available for those databases. Recommend if an upgrade is needed.
- [ ] Check the number of replicas. Create a replica from the primary if does not exists. 
- [ ] Create a dashboard and add metrics to all the databases. Suggestion: create an overview dashboard for all of them, and an specific for each one.
- [ ] Verify the backups, create a restore project and add to deadmansnith. ( I.e. we have for gitlab.com ).
- [ ] Verify if we have alerts in place.

Story points: 8.0
Improve the on-call handover and labels

Looking for feedback on the following board and labels: https://gitlab.com/groups/gitlab-com/gl-infra/-/boards/1424188?&label_name[]=SRE%3AOn-call

The premise is that all incidents will get labelled ~"SRE:On-call" by the incident template, and automatically go on this board.  Any other work which is triaged as On Call work will get tagged that way as well.  The EOC will triage those items and either remove ~"SRE:On-call" and re-label, or move it to their region.  At the end of the shift, items can be handed off to the next region, or not as the EOC sees fit.

This will work best if the ~"SRE:On-call" label is not used for ongoing work, unless that ongoing work needs to be handed off.  That means there should not be anything labelled both ~"corrective action" and ~"SRE:On-call".  We could consider using scoped tags to ensure this.

It might be better to use another tag for all of this - for example `SRE::Handoff` - and possibly `SRE::Corrective Action`

Opening it up for discussion, comments and opinions

Story points: 1.0
Monitor "autovacuum queue" on Postgres primary

The query https://gitlab.com/snippets/1889668 can be used to monitor the "autovacuum queue": the list of tables that need to be processed by autovacuum but not yet being processed + size of this list.

Goals:
1. be able to see if some tables that need processing don't get it and "wait" (we could even use some metric that is similar to "load average" and have some alerts: if we have N workers, and 2*N tables are "waiting" to be processed, it is time to trigger an alert). Here we need to know only the size of the "queue". @emanuel_ongres do you think it is easy to add it to Prometheus?
2. in case if troubleshooting is needed, it would be great to see details in logs. I usually achieve this using some cronjob with plpgsql form of that query, running every N minutes and logging details. What is the best option to do it in the case of GitLab.com infrastructure?

I think we should split the achieving of these two goals. Having item 1 is much more important. However, having item 2 can be extremely helpful as well, for the troubleshooting of autovacuum behavior.

Running this query (https://gitlab.com/snippets/1889668) makes sense only for the Postgres master. So we can wrap it to `pg_is_in_recovery()` check, not to run it on replicas, but have it everywhere.

cc @Finotto @gerardo.herzig @ahachete

Story points: 2.0
Test Postgres database restoration from GCP snapshots on staging

I'm going to test how GCP snapshots can be used for restoration of Postgres database:
- first, manually on staging (an instance with prefix `nik-` will be manually created
- next, I'll find a way to use snapshots from staging to restore to a instance in "gitlab-restore"

Additionally, an open question is: do we need pg_start_backup()/pg_stop_backup() when creating a snapshot.

Cc @Finotto

Story points: 8.0
Add alert and monitoring to ops.gitlab.net

While working on the steps to migrate ops, I noticed that ops does not have any type of monitoring or alerting with the exception of a [blackbox rule checking the sign on endpoint running](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/ops-gitlab-nets.yml) in [gprd prometheus](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/68f0e9d101ea2271a7d5a62f7e432e6d6a023d61/roles/gprd-infra-prometheus-server.json#L178). We should enable node and gitlab exporter and add some alerts.

Story points: 3.0
Terraform env gprd (and possibly others) will not plan in CI

See https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/jobs/751731 for an example.

Local planning works so this appears to be a service account permissions issue, but as the error message says, the API in question really doesn't appear to be enabled: https://console.developers.google.com/apis/library/container.googleapis.com?project=gitlab-production

Story points: 2.0
Missing permissions on gitlab-ops terraform service account

While applying the changes for [gitlab-com-infrastructure!1195](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1195), the [apply job](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/jobs/750625) returned a permissions error and failed to apply an update to the ops GKE cluster; we need to add cluster admin privileges to the environments where we're utilizing GKE

Story points: 1.0
Fix sidekiq error ratio metrics

The [sidekiq error ration panel](https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1&from=1573553266211&to=1573564066211&panelId=8&tz=UTC&fullscreen&var-PROMETHEUS_DS=Global&var-environment=gprd&var-type=sidekiq&var-stage=main&var-sigma=2) in Grafana is showing error ratios higher than 100% which can't be true and is causing alerts.

Story points: 1.0
Corrupt WAL in GKE prometheus

One of the prometheus servers in the gitlab-production GKE cluster (https://console.cloud.google.com/kubernetes/pod/us-east1/gprd-gitlab-gke/monitoring/prometheus-gitlab-monitoring-promethe-prometheus-0?project=gitlab-production) has a corrupt WAL:
`"reload blocks: head truncate failed: create checkpoint: read segments: corruption in segment /prometheus/wal/00002079 at 81103202: unexpected full record"`

AFAIK the correct way to fix that is to kick prometheus over so it can do a repair.

Story points: 1.0
Fix path to PPA in gitlab-sentry cookbook

The gitlab-sentry cookbook adds a PPA for redis, but it has a `not-if` pointing to an old `trusty` path.   This leads to add-apt-repository running every chef run, and each time it adds an additional commented out deb-src line, leading to multi-thousand line files and chef-client timeouts when it runs at busy times (add-apt-repository rewrites the file repeatedly, for some reason I don't care to investigate).

Story points: 1.0
Provision AWS IAM account for customer cross-account role access

For the request in gitlab-com/access-requests#2606, we need to provision an account for use by `gitlab.com` to [assume customer-owned cross-account IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) with permissions to provision EKS clusters into their account. At least to start, I think the `aws-account` terraform configuration in gitlab-com/gitlab-com-infrastructure> is the most likely place for this resource, though we can move/adjust if necessary at a later date.

The account should be assigned an IAM policy which allows `sts:AssumeRole` and a condition(https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html) that prevents the user from assuming a role in our own account (for now?). If we need to enable EKS provisioning, we will need to stipulate that they can not be hosted in our current production account, or make other policy adjustments to ensure that the provisioning account cannot assume other roles that grant unwanted elevated permissions.

As an example of how to negate access to roles in our production account, something like the following will be needed (change ACCOUNT_ID accordingly). (See [here](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_notresource.html#notresource-element-combinations) for additional context on why this is actually desirable in our case)

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "NotResource": [
      "arn:aws:iam::ACCOUNT_ID:role/*"
    ]
  }
}
```

Further, since we require an external ID for all cross-account roles being accessed, we _should_ be able to limit the account to only assume roles when an external id is provided, like this using the [Null](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html#Conditions_Null) condition and the [`sts:ExternalId`](https://docs.aws.amazon.com/IAM/latest/UserGuide/list_awssecuritytokenservice.html#awssecuritytokenservice-policy-keys) condition key to ensure that the permissions are only granted when that condition key is present.

```json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "NotResource": [
      "arn:aws:iam::ACCOUNT_ID:role/*"
    ],
    "Condition":{"Null":{"sts:ExternalId":"false"}}
  }
}
```

Story points: 1.0
Add about.gitlab.com redirect rule for index.html

## Problem

Crawlers are reporting duplicate content for some pages on about.gitlab.com ending with a URL `index.html`. 

Example:
- https://about.gitlab.com/analysts/forrester-vsm/
- https://about.gitlab.com/analysts/forrester-vsm/index.html

## Solution

Add a redirect rule on about.gitlab.com for any URL ending with `index.html` back to `/`.

## Concerns

@brandon_lyon @laurenbarker do you know of any templates that require their URL to end with index.html?

@gitlab-com/gl-infra/managers tagged for prioritization

Story points: 1.0
Tweak alert for disk use of /var/log on patroni servers to alert sooner

The current alert for /var/log filling up comes when 1% of the space is remaining. When /var/log is filled up, it can have negative impacts on WAL-E shipping and the node performance as a whole. We should alert a little more aggressively on the /var/log disk filling up. Perhaps at 10% remaining space.

Story points: 1.0
Make cloudflare DNS terraformable

Since we are keeping seperate zones in cloudflare for gitlab.com and staging.gitlab.com for cloudflare configuration purposes we are going to keep the split as it is right now. This means we can extend the `dns` environment in terraform to allow managing the `staging.gitlab.com.` zone via the cloudflare provider without impacting production.

This also resolves diversion from production, because even though the underlying DNS is different, there is no difference in managing it, then.

We should be able to keep the same structure for configuring that via the json files as we do for route53.

Story points: 5.0
Rails Consoles for Auto DevOps Deployments

Currently most of our rails consoles are running on VM's, and the code that they run is deployed along with all of the other nodes.

In non-core applications running from Auto DevOps, we have no way to get a rails console at the moment.

When connecting directly to an application pod, this is the result
```
$ kubectl exec -it production-5fdb8f947-5rpxq -- /bin/bash
root@production-5fdb8f947-5rpxq:/# cd /app/bin
root@production-5fdb8f947-5rpxq:/app/bin# ./rails console
Traceback (most recent call last):
	4: from ./rails:3:in `<main>'
	3: from ./rails:3:in `require_relative'
	2: from /app/config/boot.rb:3:in `<top (required)>'
	1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- bundler/setup (LoadError)
root@production-5fdb8f947-5rpxq:/app/bin# rails console
```
Also:
```
kubectl exec -it production-5fdb8f947-5rpxq -- /bin/bash
root@production-5fdb8f947-5rpxq:/# bundle exec rails c
bash: bundle: command not found
root@production-5fdb8f947-5rpxq:/# cd /app/bin/
root@production-5fdb8f947-5rpxq:/app/bin# ./bundle exec rails c
Traceback (most recent call last):
	2: from ./bundle:3:in '<main>'
	1: from /usr/lib/ruby/2.5.0/rubygems.rb:263:in 'bin_path'
/usr/lib/ruby/2.5.0/rubygems.rb:289:in 'find_spec_for_exe': Could not find 'bundler' (1.17.3) required by your /app/Gemfile.lock. (Gem::GemNotFoundException)
To update to the lastest version installed on your system, run `bundle update --bundler`.
To install the missing version, run 'gem install bundler:1.17.3'
```

If we simply deploy a VM as we do now, we would have to manually set it up and there is no mechanism in Auto DevOps to deploy code updates to that VM.

If we deploy to a pod, then we have to give full access to the cluster via `kubectl` to anyone who needs to run `kubectl exec` to get a rails console.

Neither of these are ideal.  Lets discuss options.

Story points: 1.0
Deploy a prototype of dynamic Image rescaling to CloudFlare workers on staging

Context: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8209#note_238007871

Something like this should work

```javascript
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  let url = new URL(request.url);
  let fetchParams = {
    cf: {
      image: {
        fit: 'scale-down'
      }
  };

  let width = parseInt(url.searchParams.get("width"), 10);
  if (isNaN(width) || !width) {
    return fetch(request);
  } else {
    fetchParams.cf.image.width = width;
  }

  // When we do not need the `x-with` header anymore:
  // return fetch(request, fetchParams);

  let response = await fetch(request, fetchParams);
  response = new Response(response.body, response)
  response.headers.set('x-with', 'cloudflare worker')
  return response
}

```

According to @timzallmann These are the required routes:

- User Avatar URL `https://assets.gitlab-static.net/uploads/-/system/user/avatar/1149402/avatar.png?width=24`
- Project Avatar URL `https://assets.gitlab-static.net/uploads/-/system/project/avatar/7764/about_logo.png?width=40`
- Group Avatar URL `https://gitlab.com/uploads/-/system/group/avatar/9970/logo-extra-whitespace.png`

so the `/uploads/-/system/**` pattern should be good.

Story points: 2.0
Update runbooks with documentation on how to use and manage one-off commands.

As an SRE, I need to know how to create, install, and trigger ansible based commands to help manage actions on large numbers of nodes.

The runbooks should be updated to include sections on how to accomplish each of these tasks.

This is the original issue with details on what was to be made: https://gitlab.com/gitlab-com/gl-infra/delivery/issues/75

Story points: 1.0