giseldo/neodataset huggingface dataset:
DR Redis servers have excessive load

```
load average: 30.21, 36.86, 29.86
```

On:

```
redis-01-db-dr.c.gitlab-dr.internal
redis-02-db-dr.c.gitlab-dr.internal
redis-03-db-dr.c.gitlab-dr.internal
```

Connecting:

```
$ REDIS_MASTER_AUTH=$(sudo grep ^masterauth /var/opt/gitlab/redis/redis.conf|cut -d\" -f2)
$ /opt/gitlab/embedded/bin/redis-cli -a $REDIS_MASTER_AUTH
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.251.5.102,port=6379,state=send_bulk,offset=0,lag=0
master_repl_offset:1300481
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:251906
repl_backlog_histlen:1048576
```

Story points: 1.0
Reference bootstrap/teardown scripts by module version

Currently, our [new](#6084) GCP [bootstrap module](https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/bootstrap) is an iterative improvement over versioned bootstrap script files in out legacy terraform monorepo, that still has versioned files referenced by passing an attribute to the module.

Ideally, the versioned filenames should be deprecated, and going forward, subsequent versions should be referenced by changing the module version. We need to test to make sure this is viable, and verify whether we have any cases of using multiple bootstrap versions in parallel. If so, we will have to validate using multiple instances of the bootstrap module at the same time, namespaced by version.

1. [ ] Add a copy of the latest bootstrap script to the module, _without_ a version in its filename
1. [ ] Make the `bootstrap_version` and `teardown_version` attributes optional, for referencing older versions of the bootstrap script
1. [ ] Update calling references for the module to deprecate those attributes, and switch to using the module version

/cc @T4cC0re

Story points: 2.0
Deploy Uptycs to Production

After the successful deploy to staging (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5784) next step is to roll it to production. 

To do: 

1. [x] Setup gitlab-uptycs cookbook mirroring to ops.gitlab.com.  Issue: https://gitlab.com/gitlab-cookbooks/gitlab-uptycs/issues/1  
1. [x] Add CI/CD pipeline for gitlab-uptycs cookbook.  Issue: https://gitlab.com/gitlab-cookbooks/gitlab-uptycs/issues/2
1. [x] Determine rollout plan (# roles/hosts per phase, # phases) Issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/709
1. [x] Deploy! Issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/709

Story points: 5.0
Raise GCP limits for the "restore" projects

For https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore current limits are:

- 40TiB for disks
- 96 for CPUs

For database-related tasks (such as benchmarking, testing new code, background migrations, and so on) we need production clones. When for small tasks (as new index idea verification) a smaller instance is usually enough, it is not so if we need to test something bigger (as background migration processing ~100M-1B rows). In such cases we need a 32-core machine, since it gives better disk performance (see https://cloud.google.com/compute/docs/disks/performance).

The same project is also constantly being used for automated backup verification.

Additionally, I just raised the default disk space for 'grpd' type of instances there from 3000Gib to 3300GiB, since GitLab.com is growing and the old size was not enough.

Please raise quotas. If possible, double them.

@Finotto 

cc @abrandl @northrup @yguo

Story points: 1.0
Make automatic terraform module versioning mandatory, or remove it

So this has tripped me up several times now, and I know it’s tripped up a few others as well.  The new automatic versioning for the terraform modules requires `fix` or `feat` in the MR name or it breaks and you have to manually tag it.  This is just too easy to mess up, since this is not required across the board, and many of these changes are so small and infrequent that I foresee it being a long time before I get used to doing it right.  That means a lot more messed up merges ahead.

I propose either making it impossible to merge an MR with no prefix, or removing the automated pipeline and making manual tagging the standard.

If the merge button didn't appear unless the pipeline found a prefix in the title, that would be ideal.  Then we could use `something:` as the prefix if we didn't want a version bump.

Story points: 1.0
Alerts Mis-labeled for GitalyVersionMismatch

Alerts in staging are going to production PagerDuty. In alert manager, they are labeled as `gstg-default` but in PagerDuty they are labeled as `GitLab Production`

https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitaly.yml#L173

```
  - alert: GitalyVersionMismatch
    expr: >
      count(
        sum by (environment, version) (
          gitlab_build_info{tier="stor",type="gitaly"}
        ) > 0
      ) == 2
    for: 30m
    labels:
      channel: gitaly
      pager: pagerduty
      severity: critical
    annotations:
      description: During a deployment, two distinct versions of Gitaly may be running
        alongside one another, but this should not be the case for more than 30m.
        Visit https://dashboards.gitlab.net/dashboard/db/gitaly-version-tracker?orgId=1
        for details of versions deployed across the fleet.
      runbook: troubleshooting/gitaly-version-mismatch.md
      title: 'Gitaly: two versions of Gitaly have been running alongside one another
        in production for more than 30 minutes'
```

Story points: 1.0
Enable Geo Tracking Database

The next step to enabling Geo for GitLab.com is to enable the tracking database. This is not as straightforward as the documentation suggests, so I am creating this issue to track.

The replicated main databases are set up. There is a separate instance configured for the tracking database at the secondary site.  We are at the point of running the migrations to set up the schema.

This issue can be closed when this section of the documentation is successfully completed: https://docs.gitlab.com/ee/administration/geo/replication/external_database.html#configure-the-tracking-database

cc/ @ashmckenzie

Story points: 3.0
Configure Flipper HTTP adapter on dev.gitlab.org

# Details

Now, we [can smoothly connect to GitLab Feature Flag from `Feature` class](https://gitlab.com/gitlab-org/gitlab/-/issues/222273). We should start dogfooding on a pre-production server dev.gitlab.org in order to evaluate the new architecture if it works properly

## TODO

- [ ] Announce when we reconfigure the instance. Engineers cannot update feature flags during the maintaince period.
- [ ] [Reconfigure dev.gitlab.org to use HTTP adapter](https://gitlab.com/gitlab-org/gitlab/-/issues/222266). The URL points to a GitLab Feature Flag Server (Project) in ops.gitlab.net.
- [ ] [Migrate existing flag data from local postgres to GitLab Feature Flag](https://gitlab.com/gitlab-org/gitlab/-/issues/222273).
- [ ] Announce when the migration is done. Engineers can update feature flags via chatops again.

Probably, it'd better we introduce a maintenance mode in chatops for feature flags that prevents anyone to update feature flags during the term. We're already doing the same during a production incident.

## Feature Flag Servers

- dev.gitlab.org ... https://ops.gitlab.net/gitlab-org/feature_flags/dev 
- staging.gitlab.com ... https://ops.gitlab.net/gitlab-org/feature_flags/staging
- gitlab.com ... https://ops.gitlab.net/gitlab-org/feature_flags/production

## NOTES

- Developers can change flags via Chatops or rails console (i.e. Feature.enable). You can see the strategy vs gate mapping below.
- Developers cannot change flags in GitLab's Feature Flag UI (i.e. readonly). This is because chatops has some extended features e.g. freeze all flags during a production incident. We'll likely follow this up.
- This is [a part of ~Dogfooding issues](https://gitlab.com/groups/gitlab-org/-/epics/3367).

Story points: 4.0
Put osquery Grafana dashboard under version control

We want to [cleanup manually managed Grafana dashboards](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2345). As the [osquery dashboard](https://dashboards.gitlab.net/d/fjSLYzRWz/osquery) still is needed (for monitoring the uptycs deployment, which is about to be [rolled-out with a new version](https://gitlab.com/gitlab-com/gl-security/secops/detection/uptycs/-/issues/13) by security soon), we should add it to the runbooks/dashboards folder.

Story points: 3.0
Ignore min_cpu_platform in all instances and envs in terraform

We sometimes get random changes of `min_cpu_platform` for instances in our plan. Presumably, when gcp internally decides to run a VM on a different CPU family (maybe triggered by a reboot) - that's at least my theory. This is unfortunate, because it causes an unclean plan and applying the plan needs a reboot of the instance.

```
- min_cpu_platform          = "Intel Skylake" -> null
```

As we are not interested in `min_cpu_platform` so far (we only select `machine_type`), we should make terraform ignore this attribute for all instances and envs instances.

Story points: 3.0
Planning for Observability Team W25 Milestone/Sprint

[Milestone](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/81)

## Calendar

|             | M 15 Jun | T 16 Jun | W 17 Jun | Th 18 Jun  | F 19 Jun
|-------------|----------|----------|----------|----------- |----------
| @bjk-gitlab | - | - | - | - | -
| @cindy | - | - | - | - | -
| @craigf | :pager: | :wrench: | :wrench: | :wrench: | :wrench: 
| @igorwwwwwwwwwwwwwwwwwwww | - | - | - | - | -
| @msmiley | - | - | - | - | -
| @mwasilewski-gitlab | - | - | - | - | 🧘🏻‍♂️

W25 working days: 24

|             | M 22 Jun | T 23 Jun | W 24 Jun | Th 25 Jun  | F 26 Jun
|-------------|----------|----------|----------|----------- |----------
| @bjk-gitlab | - | - | - | - | -
| @cindy | - | - | - | - | -
| @craigf | :sun_with_face: | :palm_tree: | - | - | -
| @igorwwwwwwwwwwwwwwwwwwww | - | - | - | - | -
| @msmiley | :palm_tree: | :palm_tree: | :palm_tree: | :palm_tree: | :palm_tree: 
| @mwasilewski-gitlab | - | - | - | - | -

W26 working days: 24

<hr>

**Total working days: 48**

Story points: 2.0
Update about.gitlab.com review apps prometheus probe

We have [a probe configured](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/0ce027722dac3bd7c11b8de87b3a1e8dd1d0ca1c/roles/gprd-infra-prometheus-server.json#L215) for about-src.gitlab.com. We used to serve review apps for https://gitlab.com/gitlab-com/www-gitlab-com/ from that domain, but now we do it from about.gitlab-review.app. We should update that probe.

Story points: 1.0
Move legacy redirects out of about-src to fastly

We got a [PagerDuty alert](https://gitlab.pagerduty.com/incidents/PRU4JCP) about a cert about to expire for about-src.gitlab.com. We first thought this was for about.gitlab.com but it seems it's actually for the legacy redirects we're still serving through that host (see https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/81e814df0e352d43f3c6a6e8576f4a3625346d6d/roles/about-gitlab-com.json#L7-23). We should move those to our [terraform redirects environment](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/tree/master/environments/redirects) which will put them on Fastly and manage SSL certificates through it.

- [x] https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1784
- [x] https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3606
- [x] `knife vault edit about-gitlab-com _default` and remove the old SSL certs

/cc @ahmadsherif @ggillies FYI in case you saw that alert.

Story points: 1.0
[Runbook] update the runbook to create or restore a delayed/archived replica

Review the existent runbooks, and update the process on the PostgreSQL version 11.7. 

Also update the reference and commands for the usage of wal-g to apply the wals.

In these environments, PostgreSQL is installed by omnibus.

Please consider the case when the replica is out of sync and how to troubleshoot and fix the instance.

Story points: 3.0
Completely decommission uptycs from our fleet

Security wants to rollout a new version of uptycs. To prevent any glitches, uptycs is recommending to completely remove the previous version from the fleet first: https://gitlab.com/gitlab-com/gl-security/secops/detection/uptycs/-/issues/13#note_349679029

We need to update the `gitlab-uptycs` cookbook to support de-installing the package and deleting the data dir and then execute the decommissioning of the old uptycs version.

Story points: 5.0
postgres checkup does not include information from database primary

The latest postgres-checkup report does not seem to include information about the primary database instance: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10326

I suspect this is only a configuration issue with the pipeline (but don't have access). It would be helpful to have the primary included, particularly to understand the read/write workload better.

The project to look at is: https://ops.gitlab.net/gitlab-com/gl-infra/postgres-checkup I recall we've had a list of database instances to connect to in the CI config, but I'm not sure if that's still true.

cc @albertoramos

Story points: 3.0
Setup CI pipelines for db ops automation

With new database ops automation code living in https://ops.gitlab.net/gitlab-com/gl-infra/db-ops we want to have CI pipelines setup to run different database maintenance tasks automated via CI. This should also help us to integrate with chat-ops in the future.

First iteration:
* Run [rolling postgres restarts on replicas](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10333) via CI

Story points: 5.0
Create Ansible playbooks for postgres restart and failover

For the [planned DB failover](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10276) we want to have Ansible playbooks in order to make this task automated and repeatable.

The playbooks should live in the new db-ops repository: https://ops.gitlab.net/gitlab-com/gl-infra/db-ops

First iteration:

* automate rolling postgres restart on (selected) replicas
* automate primary failover

Story points: 8.0
Redis Cache Sentinel hosts should not be labelled as `type=redis` in Prometheus

Currently, the Redis Cache Hosts are labelled in redis as part of the `redis` service. They should be part of the `redis-cache` service.

https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=count(up%7Btype%3D%22redis%22%2C%20env%3D%22gprd%22%7D)%20by%20(fqdn)&g0.tab=1

Story points: 1.0
rebuild patroni-02

Patroni-02 failed and was taken out of the cluster. In order to align the patroni cluster nodes properly by number again to be able to [decommission 4 other nodes](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1741), we need to re-build patroni-02 from scratch (forcing it to be instantiated on a different gcp hardware node hopefully).

We need to rebuild patroni-02 before we do the [planned primary switchover](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10276).

Story points: 3.0
Remove overly broad page rule from Cloudflare configuration

Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2145

[This page rule](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/blob/master/environments/gprd/main.tf#L2463-2475) should be removed from our configuration.

The global security settings should be considered our final fall through page rule. Global security and caching settings should be made at the zone level and not with a page rule.

Story points: 1.0
Create (or modify) a runbook to describe how to identify authenticated vs unauthenticated API calls

Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2145

As an on-call SRE, I should be able to identify and characterize API traffic, including if that traffic is authenticated or unauthenticated. And, if possible, programmatically isolate this traffic to better craft page rules and defenses in Cloudflare (or Haproxy) to protect our site from abuse.

Story points: 2.0
Create (or update) Cloudflare runbook to better address abuse and attack events

Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2145

As an on-call SRE, I should have a runbook that helps describe how to identify an abuse or attack in Cloudflare, and describe the tools we can use in Cloudflare to mitigate such an event.

This should probably include sections on:
1. Creating specific page rules to over-ride our security level for a URI
1. Adding IP addresses to a block list
1. Changing the zone-wide security level
1. Explain how the global `Im under attack` toggle will affect the site

Story points: 2.0
Plan patroni failover to shrink the cluster (and more)

We want to execute some changes to our production patroni cluster which require a short downtime for a failover:

* shrink the cluster size: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1741
* increase max_connections from 300 to 500
* Reconfigure our Terraform template
* Anything else?

We need to plan when and how to execute this change, evaluating the risk and impact of a failover and which actions are needed to cleanup after the failover.

Story points: 1.0
Fix creation of wal-g gcs.json on grpd postgres replicas

During execution of https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2161, chef created an empty `/etc/wal-g.d/gcs.json` file instead of filling it with data.
I copied over the file from wal-e to fix this for now (and confirmed it is not overwritten by chef), but we need to get this fixed in chef.

Story points: 3.0
Adjust node disk IO quota metrics to be per node, not per device

As mentioned in the observability meeting, it turns out that GCP disk IOP quota is per node, sum by the total HDD and SSD disk size per node. Not IOPs per disk per node.

We need to adjust the automatic calculations for these metrics to calculate per disk.

We also need to have info metrics to identify devices by HDD and SSD so that we can calculate per-node saturation.

Story points: 1.0
Alert when jobs are not being processed by sidekiq

Corrective action for:
* incident https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2154
* RCA https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2158



To avoid the situation where we are unaware of sidekiq queues that are not being processed at all we should implement alerting on low RPS that will be useful so that we can page the oncall for similar problems as in the linked incident. As a first iteration it would be good to have a very generous low threshold.

@andrewn suggested in slack:

> We alert on any job that maintains a minimum 0.1 rps over the course of the day. If we don’t see it for 6 hours, we alert.
> Obviously this will we a bit noisy when queues are decommissioned

Story points: 3.0
Clean up unused deployments in gs-staging cluster

The `gs-staging` cluster has a bunch of failed deployments, and deployments related to projects that no longer exist.

```
version-gitlab-com-6491770-dast-default
version-gitlab-com-fork-12446120-review-sync-upstr-vthqx5
version-gitlab-com-fork-12446120-staging
license-gitlab-com-fork-12446131-review-enable-aut-a96d5t
license-gitlab-com-fork-12446131-dast-default
license-gitlab-com-fork-12446131-staging
```

Story points: 1.0
Node provisioning broken due to gcloud gem install failure

The ruby version bundled with td-agent is 2.4, now when we install the `gcloud` gem it is complaining that that `google-protobuf` requires ruby 2.5.


```
* Mixlib::ShellOut::ShellCommandFailed occurred in chef run: execute[install gcloud gem] (gitlab_fluentd::default line 170) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /usr/sbin/td-agent-gem install gcloud --no-document -v 0.24.0 ----
STDOUT:
STDERR: ERROR:  Error installing gcloud:
	google-protobuf requires Ruby version < 2.8.dev, >= 2.5.
---- End output of /usr/sbin/td-agent-gem install gcloud --no-document -v 0.24.0 ----
Ran /usr/sbin/td-agent-gem install gcloud --no-document -v 0.24.0 returned 1

```


One a working node:

```
google-protobuf (3.11.3 x86_64-linux, 3.9.0 x86_64-linux, 3.6.1 x86_64-linux)
```

Installing `google-protobuf` manually seems to work around this 

```
/usr/sbin/td-agent-gem install google-protobuf --no-document -v 3.11.3

```



```
# /usr/sbin/td-agent-gem install google-protobuf --no-document -v 3.11.3
Fetching: google-protobuf-3.11.3-x86_64-linux.gem (100%)
Successfully installed google-protobuf-3.11.3-x86_64-linux
1 gem installed

```

Story points: 2.0
gitlab-uptycs cookbook is not stopping osqueryd service

When setting `"enabled": false` in the [chef role](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3454) for gitlab-uptycs, the osqueryd service on all affected nodes should be stopped and disabled, but that isn't working.

This MR also didn't fix it: https://gitlab.com/gitlab-cookbooks/gitlab-uptycs/-/merge_requests/22

Story points: 3.0
Decomission vfiles module and revert any bootstrap migration changes

This [EPIC](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/206) outlined a series of steps to alter how our bootstrap process worked for chef in GCP. That work is on hold due to another option being put in place and future terraform changes breaking this new approach.

We should remove the vfiles module from all environments except ci-org.

Story points: 1.0
Fix patroni installation in Chef

When building new patroni nodes, after Chef has run, `gitlab-patronictl` is missing dependencies:

```
root@patroni-08-db-gprd.c.gitlab-production.internal:~# gitlab-patronictl list
Traceback (most recent call last):
  File "/opt/patroni/bin/patronictl", line 7, in <module>
    from patroni.ctl import ctl
  File "/opt/patroni/lib/python3.5/site-packages/patroni/ctl.py", line 29, in <module>
    from patroni.config import Config
  File "/opt/patroni/lib/python3.5/site-packages/patroni/config.py", line 12, in <module>
    from patroni.postgresql.config import ConfigHandler
  File "/opt/patroni/lib/python3.5/site-packages/patroni/postgresql/__init__.py", line 3, in <module>
    import psycopg2
ImportError: No module named 'psycopg2'
```

We need to fix that in Chef.

Story points: 1.0
Decommission left-over patroni nodes

During the postgres 11 upgrade we left patroni-{08,09,10,12} out of the cluster and on version 9.6 to be able to roll back.
As the cluster seems to be running fine without those nodes and we constantly need to silence alerts for those nodes and they also are quite expensive, we should commission them ASAP.

The current primary is patroni-11, which makes decommissioning more complicated, as we only can savely remove nodes in terraform starting with the highest number. This means, we probably need to rebuild and add patroni-08 to the cluster, failover to one of the lower-numbered nodes, remove patroni-11 from the cluster and then decommission patroni-[09-12].

Story points: 5.0
Observability Team - W23 2020 | Sprint Planning Issues

# Planning - W23 2020

We'll use this issue as a means of discussing the priorities for the [Observability Team - W23 2020](https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/80) milestone.

![Team_Focus](/uploads/655795e3b83206641ca3a4b2f5e7b167/Team_Focus.jpeg)

## References

- [~"team::Observability" Epics Roadmap](https://gitlab.com/groups/gitlab-com/gl-infra/-/roadmap?label_name%5B%5D=team::Observability&scope=all&sort=end_date_asc&state=opened&utf8=%E2%9C%93&layout=QUARTERS)

Story points: 3.0
The `testbed` environment appears to be routing alerts to production destinations

Alerts in the `testbed` environment are being send to production slack channels, leading to false alerts. 

Example: https://gitlab.slack.com/archives/CD6HFD1L0/p1589276190114400

<img src="/uploads/933b8888bc9e3be37a5c2ab068ffb40a/image.png" width="500">

cc @T4cC0re oncall @AnthonySandoval

Story points: 1.0