giseldo/neodataset huggingface dataset:
Setup/update Terraform Admin Project

Following a slight modification of the pattern [here](https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform#add_organizationfolder-level_permissions), I propose utilizing the `env-zero` project as our central place to manage project permissions and service accounts for automation under #6810. @devin is already doing the same for https://ops.gitlab.net/gitlab-com/group-projects.

As for the modification, instead of using a single admin-level service account to provision resources in _all_ projects, we should create a service account for each project (environment), with permissions scoped to allow access to resources only within that project. Once a project is bootstrapped, the relevant credentials are then added to the environment-specific CI variables for automated terraform runs within that project. However, I'm still not sure if it is possible to have the project-specific service accounts in the terraform admin project, or within the projects they will be used to manage.

This issue is to track research/discovery and effort to implement this type of setup, including documentation and/or [a bootstrap script](https://github.com/monterail/terraform-bootstrap-example) to work around circular dependencies so that this project can also be managed in terraform, along with (possibly) relocating and consolidating the state files for all managed child projects (same questions/rationale as the point above about where to scope the project-specific service account).

/cc @gitlab-com/gl-infra FYI for discussion

Story points: 2.0
Postgres-related services nearing capacity in CaPlan dashaboard

Postgres-related services are *nearing capacity* on the [Caplan dashboard](https://dashboards.gitlab.net/d/TeJU3AIWz/capacity-planning?orgId=1):

![Screen_Shot_2019-07-30_at_7.47.34_PM](/uploads/f4075aab00becd7c66ba705f1055d97b/Screen_Shot_2019-07-30_at_7.47.34_PM.png)

Please investigate, and either fix the dashboard or create issues to address specific caplan concerns. Ongres should be aware of how the caplan metric is calculated so they can help.

Link said issues to this issue and tag with `Obserbavility` and `caplan`.

Story points: 2.0
Move gstg back to wal-e

We decided to discontinue testing wal-g in staging because it hinders our ability to test changes to wal-e before we release them to production.

Background: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7359

> From a high level perspective, we transitioned over to use wal-g in staging in the beginning of 2019. We hoped to be able to use it in production too, to alleviate the pressure on the primary (wal-g is capable of taking basebackups off of a secondary, wal-e is not). However, we kept running into wal-g issues (with GCS in particular) and kept waiting on wal-g releases and so on. wal-g never made it to production because of that.

The task here is to actually move gstg back to use wal-e:

* [x] Reconfigure gstg for wal-e
* [x] Deploy change (daily backup, WAL push)
* [x] ~~Resync DR replicas (they will not like this, we'll just resync with a (base)backup and use wal-e going forward).~~ Not needed as the transition was gapless. Letting them catch up now.
* [x] Have backup testing project use wal-e for staging, too

Story points: 2.0
Use native instrumentation for camoproxy

go-camo now has native Prometheus support.

* [x] Enable monitoring in Prometheus scrape config.
* [x] Cleanup textfile exporter.
* [x] Remove obsolete `--stats` flag from gitlab-camoproxy cookbook.
* [x] Add metrics to upstream go-camo to replace mtail metrics.
* [x] Remove mtail watcher.
* [x] Create Grafana dashboard.

Story points: 3.0
Right-size redis persistent persistent and redis-sidekiq nodes

The redis-sidekiq nodes were sized the same as the previous redis-persistent nodes, as n1-standard-8, for safety during the initial migration.   This gives them 8 CPU and 30GB of RAM

All of our redis nodes generally need no more than 2 CPU (1 for redis, 1 for other things); the sizing is for RAM.

Redis on the redis-sidekiq nodes is currently using <1GB RES + VIRT, and another 7GB of general cache.  8GB RAM total would be more than enough.  And n1-standard-2 should be sufficient.

On the old redis persistent nodes, the running system is using about 9GB with 12GB of general cache.  We might be able to get away with 15GB RAM for this, but I'm less confident, and would like to see some more usage before deciding that.

Story points: 2.0
GitLab.com Performance Metric

Per https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#gitlab-com-performance, we need to produce a performance metric that _reflects the performance of GitLab as experienced by users_. Something along these lines was outlined in the [Service Levels and Error Budgets](https://about.gitlab.com/handbook/engineering/infrastructure/blueprint/service-levels-error-budgets/) blueprint.

Let's do a first (if possibly rough) iteration, whether with Pingdom or Thousand Eyes. In particular, we can start using the [SLIs outlined in the blueprint](https://about.gitlab.com/handbook/engineering/infrastructure/blueprint/service-levels-error-budgets/#slis).

Please update the Infrastructure Performance Indicators handbook page to reflect the health, maturity and URL for the data.

Story points: 3.0
postgres-dr-delayed-01-db-gstg replication is broken

```
Alertname: PostgreSQL_ReplicationLagTooLarge_DelayedReplica
 Channel: database
 Env: gstg
 Environment: gstg
 Fqdn: postgres-dr-delayed-01-db-gstg.c.gitlab-staging-1.internal
```

```note: Replication lag on server postgres-dr-archive-01-db-gstg.c.gitlab-staging-1.internal:9187 is currently 9d 2h 7m 11s```

From the logs:
```
2019-07-26_00:18:07.60831 ERROR: 2019/07/26 00:18:07.608213 Archive '0000006200003C7500000073' does not exist.
2019-07-26_00:18:07.93544 ERROR: 2019/07/26 00:18:07.935350 Archive '00000063.history' does not exist.
```

Story points: 1.0
Enable external merge request diff storage

On GitLab.com, `merge_request_diff_files` is the biggest table at about 1.1TB currently and it grows at a rate of about 60 GB per month.

![Screenshot_from_2019-07-25_18-59-00](/uploads/cb49c0e72b8b4ca62dbdee6a27ffc5a3/Screenshot_from_2019-07-25_18-59-00.png)

We should look into enabling external merge request diff storage as pointed out below: https://docs.gitlab.com/ee/administration/merge_request_diffs.html

### Update (2020-02-26):

The current size of the relevant table `merge_request_diff_files` is 1,659 GB out of 5,400 GB total database size (about 30% of the total database size is this single table). Once the migration is done, all of this data should live outside the database and reduce the database size by 30%.

Story points: 5.0
Delete performance.gitlab.net cert in sslmate

It just auto renewed:
```
From: SSLMate <support@sslmate.com>
To: ops-notifications@gitlab.com
Subject: SSLMate Certificates for performance.gitlab.net


Your certificates for performance.gitlab.net are ready!
```
but we don't use it anymore (replaced by dashboards.gitlab.net).  We need to stop auto-renewing it and let it expire gracefully

Story points: 1.0
imap_mailbox_exporter not working

While working on gitlab-cookbooks/gitlab-exporters!97 I encountered the `gitlab-exporters::imap_mailbox_exporter` recipe failing on my kitchen setup. Unable to figure out why, I went to a production mailroom box to encounter it had the same issue:

```
2019-02-15_13:37:37.07491 time="2019-02-15T13:37:37Z" level=info msg="Exporter listening on 0.0.0.0:9117" source="imap-mailbox-exporter.go:194"
2019-02-15_13:37:37.07501 time="2019-02-15T13:37:37Z" level=fatal msg="listen tcp 0.0.0.0:9117: bind: address already in use" source="imap-mailbox-exporter.go:196"
```

The production log at `/var/log/prometheus/imap_mailbox_exporter/current` on mailroom-02-sv-gprd.c.gitlab-production.internal also has some other errors, the point being that the exporter is not serving data (`curl localhost:9117` just hangs)

/cc @ahmadsherif since I see your name in that recipe's code

Story points: 2.0
Anomaly-based detection of sidekiq queue size

Following from https://gitlab.com/gitlab-com/gl-infra/production/issues/992 we only alert on some specific sidekiq_queue_size labels, or generically if a given queue exceeds 50000.

It seems to me that we could use some of the anomaly detection techniques already established (recording rules in `rules/service_ops_rate.yml`, alerts in `rules/general-service-alerts.yml`) to be more flexible about noticing unusual volumes of queuing (such as we had in the incident), rather than trying to manage hard-coded limits.

Story points: 3.0
Update runit cookbook to v5.x

One of the errors we encountered [while testing chef-client 15](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7258) was with the `runit` cookbook. We should update to v5.x of it. Notice that according to their CHANGELOG versions >= 5.0 require at least chef 14 https://github.com/chef-cookbooks/runit/blob/master/CHANGELOG.md#500-2019-02-04. This means that the cookbook updates for these updates will need to held for deployment until we roll out chef 15.

According to chef-repo's Berksfile.lock, the following cookbooks are pinned to runit ~> 4.3 and will need update:

- gitlab-alertmanager: https://gitlab.com/gitlab-cookbooks/gitlab-alertmanager/merge_requests/40
- gitlab-camoproxy: https://gitlab.com/gitlab-cookbooks/gitlab-camoproxy/merge_requests/10
- gitlab-elk: https://gitlab.com/gitlab-cookbooks/gitlab-elk/merge_requests/111
- gitlab-exporters: https://gitlab.com/gitlab-cookbooks/gitlab-exporters/merge_requests/97
- gitlab-monitor: https://gitlab.com/gitlab-cookbooks/gitlab-monitor/merge_requests/27
- gitlab-mtail: https://gitlab.com/gitlab-cookbooks/gitlab-mtail/merge_requests/45
- gitlab-prometheus: https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/merge_requests/458

Ideally in those MRs we add rspec examples if missing, and include the recipes that use `runit_service` in kitchen to make sure the update didn't break them.

Part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6942

/cc @ahanselka @nnelson @devin @cmcfarland @dawsmith

Story points: 4.0
Plan In lieu of Automating Access Request issues

Access Requests are a source of ~toil for the SRE on-call and SRE managers. Issues are manually ingested and approved blindly. After a requestor's manager approves the issue, SRE managers have no way of knowing whether the requestor should or should not be granted access–nor should we. If these issues are used as an audit trail, I'm no longer comfortable adding my approval. And if we should continue processing without challenge, then I propose we remove the below language altogether.

> INFRASTRUCTURE MANAGER: For requests involving access to critical Infrastructure systems, an additional layer of approval is required. Review the requests and, if approved, copy and paste `/label ~"InfrastructureApproved"`

Longterm, I expect we will roll out role-based access using OKTA and IAM. But, regardless of the outcome of the aforementioned approval process, an SRE's focus should not be hampered with access requests. Short of fully automating the workflow, I advocate for one of the following options.

1. Process requests in batch by type at the start of each week.  
_or_  
2. Document the process and instruct requestors to submit merge requests to the cookbook.


@gitlab-com/gl-infra/managers 

Cc @glopezfernandez

Story points: 2.0
The latest version of the prometheus-operator breaks during upgrade

Due to not locking down the version of our component: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7110

In the `pre` environment, we run into the following (this was from testing locally):

```
UPGRADE FAILED
Error: Deployment.apps "gitlab-monitoring-kube-state-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"kube-state-metrics"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Error: UPGRADE FAILED: Deployment.apps "gitlab-monitoring-kube-state-metrics" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app.kubernetes.io/name":"kube-state-metrics"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
Stopping Tiller...
Error: plugin "tiller" exited with error
25s monitoring master● % helm search stable/prometheus-operator
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                               
stable/prometheus-operator      6.1.0           0.31.1          Provides easy monitoring definitions for Kubernetes servi...
56s monitoring master● %
```

This is an upgrade from version 5.15.0.  (currently running on gstg)

Use this issue to figure out what is going on and what we need to accomplish to cleanly upgrade the `gstg` without downtime.  This is to be utilized as a practice run to determine how we can investigate and perform clean upgrades between versions of a component that we do not manage.

/cc @gitlab-org/delivery

Story points: 1.0
Google network failure detection

We seem to have had a number of failures on Google's network recently. During the RCA Deep Dive for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7314, @andrewn brought up the idea of investigating how to determine the health of the network, perhaps by looking at an aggregate of our network traffic and looking for significant deviations. IF nothing else, this will help us better troubleshoot and track root causes.

Story points: 2.0
Make GKE node pools Production Ready

I’ve noticed a few things we’re going to have to think about with using terraform to manage K8s. Mainly that we’re probably going to have to use multiple node pools, if not multiple clusters for production instances.  

Any change to the nodes (disk space, etc.) results in the entire node pool being deleted and re-created at once, including all running pods.  It doesn’t do it in a rolling way like you’d expect.

This issue is to track investigating whether there is a way to make it roll updates, or whether we need to find a workaround so that we can run these changes with no downtime.

/cc @skarbek

Story points: 5.0
Availability: MTTR Metric => 90%

Define, develop and track `Mean-Time-to-Recover` (MTTR) metric as a KPI for  Infrastructure as a department. Iterate as necessary.

Update URL, Health, Maturity and Next Steps on the Handbook's [Infrastructure Performance Indicators](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#mean-time-to-recovery-mttr) page as necessary.

Epic : https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/88

Story points: 2.0
Observability: MTTD Metric

Define, develop and track `Mean-Time-to-Detect` (MTTD) metric as a KPI for  Infrastructure as a department. Iterate as necessary.

Update URL, Health, Maturity and Next Steps on the Handbook's [Infrastructure Performance Indicators](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#mean-time-to-detection-mttd) page as necessary.

Story points: 2.0
Reliability: MTBF Metric => 90%

Define, develop and track `Mean-Time-between-Failures` (MTBF) metric as a KPI for both Infrastructure as a department and Site Reliability Engineering as a team. Iterate as necessary.

Update URL, Health, Maturity and Next Steps on the Handbook's [Infrastructure Performance Indicators](https://about.gitlab.com/handbook/engineering/infrastructure/performance-indicators/#mean-time-between-failures-mtbf) page as necessary.

Story points: 2.0
Rebuild patroni-06 instance

After https://gitlab.com/gitlab-com/gl-infra/production/issues/948 has closed out, we're left with a shut down and otherwise unused patroni-06 instance.

We should clean up and do one of the following:

1. Rebuild patroni-06 and remove patroni-07 from the cluster
1. Just terminate/delete patroni-06, leaving back a "gap" in the instance name sequence (does TF cope with that well?)

Story points: 2.0
More CPU for sidekiq-pipeline nodes?

updated:
There have been some intermittent reports lately of pipelines not running in a timely fashion.  I believe this is due to CPU contention on the pipeline sidekiq nodes at peaks, and we should consider giving those nodes more CPU.

Original:
There have been some intermittent reports lately of pipelines not running in a timely fashion.  I believe this is due to increased load on the pipeline sidekiq nodes, and we may have to expand this group from 3 to 4, or more.

Story points: 2.0
DR Site Patroni won't stay replicated

The runbook to resynchronize the DR site database from the master using WAL replication works.  It takes a day or so, but it ends up replicated.  https://gitlab.com/gitlab-com/runbooks/blob/master/howto/geo-patroni-cluster.md

However it has never stayed synchronized.  We have to manually resync it every week or two. This will not work when we go live.  It needs to stay replicated on its own.

This time, it looks like something going on with Patroni:

```
+---------------+---------------------------------------+--------------+------+----------+----+-----------+
|    Cluster    |                 Member                |     Host     | Role |  State   | TL | Lag in MB |
+---------------+---------------------------------------+--------------+------+----------+----+-----------+
| pg-ha-cluster | patroni-01-db-dr.c.gitlab-dr.internal | 10.251.9.101 |      | starting |    |   unknown |
| pg-ha-cluster | patroni-02-db-dr.c.gitlab-dr.internal | 10.251.9.102 |      | starting |    |   unknown |
| pg-ha-cluster | patroni-03-db-dr.c.gitlab-dr.internal | 10.251.9.103 |      | starting |    |   unknown |
+---------------+---------------------------------------+--------------+------+----------+----+-----------+
```

@Finotto @dawsmith @abrandl - can we get Ongres or someone more familiar with our database set up to take a look at this and make some recommendations?

Story points: 4.0
Cleanup DNS Information in Runbooks

We have some stale runbooks that reference PowerDNS, which is apparently no longer in use (discovered in gitlab-com/gl-infra/infrastructure#2332).

We need to move https://gitlab.com/gitlab-com/runbooks/blob/master/howto/internal_dns.md to simply `dns.md` and include information about external DNS, too (see @dawsmith's comment in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/3004#note_193464998).

Story points: 1.0
RCA Deep Dive of gitlab-com/gl-infra/infrastructure#7090

@abrandl and @cmcfarland we'll perform a walk through of the incident during the DNA meeting on 24 Jul. Please re-familiarize yourselves with the incident and RCA. This issue has much more to do with process than troubleshooting. It's a great example, illustrative of successful collaboration between Development and Infrastructure, overlapping on-calls for DBRE and SRE, and well executed corrective actions. Also, with the timezone split, this issue can be covered in both the EMEA and Americas timezones!

Story points: 1.0
Proposed Label Cleanup

I've cleaned house and deleted some really old labels with < 10 issues, most or all of which were closed with updates longer than one year ago.

My list of remaining issues I'm less sure about are in the table below.

|Label Name | Total Number of Issues | Remove |Reason|
|-----------|-----------------------:|-------:|------|
|Bronze     |                       6|       Y||
|moved 1    |                     203|       Y|| 
|moved 2    |                      74|       Y||
|moved 3    |                      33|       Y||
|moved 4    |                      17|       Y||
|critical   |                     118|       Y|Captured by priority labels|

Story points: 2.0
Kubernetes Deploys perform a set-it and forget-it style so we don't see failures

When testing an upgrade of a Kubernetes Application, the deployment went through just fine, and helm thought the deployment was successful, however, the new replicaset that was created was not coming up properly.  The pod that was starting was stuck in a crashloop.  With this use case the deployment was technically not successful and required a roll back.  It would be wise to add something to our deployment pipeline to detect this failure and fail the pipeline in instances such as this.

Story points: 3.0
Version of components not exposed in prometheus when deployed into Kubernetes

Currently we don't have visibility via our metrics to which version of a component is running in Kubernetes.  We are running version 2.7.1 of the container registry, but I cannot seem to find this anywhere.  This would be useful in finding metrics specific to version changes, watching upgrades and roll backs run on clusters, and increase the visibility into what is running at any given time.

Story points: 3.0
On a failed registry upgrade/downgrade we should alert if one of the pods fails to come up

Downgrading the registry to version 2.7.0:

can we alert on kube_pod_status_ready?

```
$ k get pods -n gitlab
NAME                                                   READY   STATUS             RESTARTS   AGE
gitlab-certmanager-788c6859c6-zk25p                    1/1     Running            0          5d23h
gitlab-issuer.4-lx655                                  0/1     Completed          0          2m45s
gitlab-nginx-ingress-controller-78fb4c686b-d8s5t       1/1     Running            0          14m
gitlab-nginx-ingress-controller-78fb4c686b-nd5lg       1/1     Running            0          7d
gitlab-nginx-ingress-controller-78fb4c686b-zk6r6       1/1     Running            0          14m
gitlab-nginx-ingress-default-backend-7f87d67c8-blzzq   1/1     Running            0          6d15h
gitlab-nginx-ingress-default-backend-7f87d67c8-gmt7l   1/1     Running            0          14m
gitlab-registry-68cbc8c489-7ftcj                       1/1     Running            0          14m
gitlab-registry-68cbc8c489-8n2wm                       1/1     Running            0          14m
gitlab-registry-7f64787dd6-htttg                       0/1     CrashLoopBackOff   4          2m45s
```

Story points: 3.0
Schedule RCA Deep Dives

Please schedule RCA deep dives for DNA meetings. To do so:

* create an issue for a deep dive,
* assign to a team member
* assign a due date to match a DNA meeting
* link the relevant production incident and incident (RCA) issues
* link the deep dive issue to the following Epic: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/83

Story points: 1.0
When starting from an empty cluster, k-ctl does checks for objects in a non-existent namespace

One of the checks that `k-ctl` validates is that we've got our secrets already loaded into an environment.  https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/blob/master/bin/k-ctl#L47

This doesn't work out so well if we are attempting to install something for the first time on a completely clean cluster.  One, the namespace does not yet exist, and we fail to bail out of the script at this point.  Secondly, in order to install, we need the certificates.  However, the namespace is installed by helm.  

To work around this we need to skip the validation part.  And/or we can create the namespace, drop in the secrets, then perform the install.

This is not documented, and is a bit confusing.  Utilize this issue to come up with a better way to handle this cyclic dependency.

/cc @gitlab-org/delivery

Story points: 1.0
Logrotate on ops.gitlab.net is broken

I got a page today for `ops.gitlab.net` being at 1% disk space. Upon investigation some of the logs had data back from last year. 

We don't seem to be sending these logs anywhere so I didn't want to just delete them all. Because `production.log` is also in `production_json.log`, I cleared out `production.log`.

```
root@gitlab-01-inf-ops.c.gitlab-ops.internal:/var/log/gitlab/gitlab-rails# >production.log
```


 We need revisit log retention and fix logrotate.

Story points: 3.0