giseldo/neodataset huggingface dataset:
Adjust last backup alert threshold to a meaningful value

We currently alert if the last successful basebackup was more than 48h ago. This is problematic as it will randomly alert either around 14h after a backup failed (if the next backup took slightly longer than the last successful one) or not alert at all (if the next backup took less time then the last successful one). We should set it to something like 30h to give a 4h hour tolerance if the backup takes unexpectedly longer then the previous one for some reason (e.g. the Sunday backup might finish slightly faster then the Monday backup because of less traffic).

Looking at [thanos](https://thanos-query.ops.gitlab.net/graph?g0.range_input=2w&g0.max_source_resolution=0s&g0.expr=gitlab_com%3Alast_wale_successful_basebackup_age_in_hours&g0.tab=0), the variance in backup time is less than 2h.

Story points: 1.0
Patroni not failing over when data disk is full

We had an [incident in staging](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2583) where the data disk on the primary was running full, making the DB unavailable, but Patroni did not fail over to another node.

The patroni logs contained these exceptions all the time:

```
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  2020-08-27 11:50:01,650 ERROR: get_postgresql_status
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  Traceback (most recent call last):
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 505, in query
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      with self.patroni.postgresql.connection().cursor() as cursor:
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/postgresql/__init__.py", line 222, in connection
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      return self._connection.get()
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/postgresql/connection.py", line 23, in get
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      self._connection = psycopg2.connect(**self._conn_kwargs)
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/psycopg2/__init__.py", line 130, in connect
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  psycopg2.OperationalError: could not connect to server: Connection refused
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  #011Is the server running on host "localhost" (127.0.0.1) and accepting
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  #011TCP/IP connections on port 5432?
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  During handling of the above exception, another exception occurred:
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  Traceback (most recent call last):
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 452, in get_postgresql_status
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      self.server.patroni.postgresql.lsn_name), retry=retry)[0]
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 424, in query
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      return self.server.query(sql, *params)
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:    File "/opt/patroni/lib/python3.5/site-packages/patroni/api.py", line 511, in query
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:      raise PostgresConnectionException('connection problems')
2020-08-27_11:50:01 patroni-01-db-gstg patroni[24887]:  patroni.exceptions.PostgresConnectionException: 'connection problems'
```

We need to make sure that patroni is able to fail over when the primary becomes unavailable because of no space left on the data disk.

Story points: 5.0
Prevent DB WAL files to fill up the data disk

Failing `archive_command`, unused replication slots or [failing logical replication](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2583#note_403884088) can prevent WAL files to be cleaned up. We need to take measures to prevent them to fill up the data partition as this can lead serious problems and data loss.

Story points: 3.0
Alert on reboots

While alerting on symptoms should be preferable, for some classes of nodes we can be certain that a reboot will cause noticeable issues. It would be helpful to get an alert for a reboot in this case, because often from looking at the symptoms it is not always easy to deduce a reboot as cause and sometimes the symptoms might still be below thresholds so that we would miss frequent reboots.

We should alert (but probably not page) for reboots of Gitaly and Patroni nodes.

We should also alert on high reboot frequencies for all nodes.

There is a reboots dashboard already: https://dashboards.gitlab.net/d/yzukVGtZz/reboots?orgId=1

Story points: 3.0
GitLab connection troubleshooting instructions for customers

When customers are reporting problems connecting to GitLab.com it is often hard to debug the cause because of missing information. We should provide troubleshooting instructions for customers so they can help us debug the problems and make sure support has documentation at hand to point customers to.

A basic first aid kit would be something like

* `traceroute gitlab.com`
* `curl http://gitlab.com/cdn-cgi/trace`
* `curl https://gitlab.com/cdn-cgi/trace`
* `curl -svo /dev/null https://gitlab.com`

For the future, we should work on enabling [NEL](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10754).

Story points: 1.0
[Runbook] Update the permanent maintenance mode section of the patroni management runbook

During the "simulation" demo, it was noticed that there was an area or two that could be improved.

Update the permanent maintenance mode section of the patroni management runbook so that `Step 3` includes an explanation of why, not just what: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/patroni-management.md#permanently-marking-a-replica-as-in-maintenance

Story points: 2.0
Improve postgres runbook

I recently received a page for increasing numbers of dead tuples on `patroni-01`, and ran into a dead end while [following the runbook to investigate](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/postgres.md#tables-with-a-large-amount-of-dead-tuples). As a follow-up to the investigation from https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2506, we should work with Ongres to ensure that the runbook leads to better/actionable steps for resolution.

/cc @albertoramos @Finotto

Story points: 1.0
Install `pg_activity` observability tool on the patroni fleet

**Current Situation**
<!--Describe the current situation which you are intending to iterate on with the work described in this issue.-->

Currently in order to inspect activity in the database one must open a `psql` session using `sudo gitlab-psql` on a particular node and run a query: `SELECT * FROM pg_stat_activity;` and then page through results until one finds what one is looking for.

**Desired Outcome**
<!--Describe the intended outcome of the work represented in this issue. How does it improve upon the current situation above?-->

This change proposes that we install the `pg_activity` tool: https://wiki.postgresql.org/wiki/Monitoring#pg_activity


Project: https://pypi.org/project/pg-activity/

Source homepage: https://github.com/dalibo/pg_activity

This will support an easier method for examining database activity with a wider range of options for sorting and filtering that are only a keystroke away, instead of crafting or copypasta queries into a database `psql` session.

**Acceptance Criteria**
<!--List out work items necessary to arrive at the desired outcome above, as atomically as possible.-->
- [x] The json chef role for the staging patroni fleet will have to be modified to include:
   1. [x] The installation of the `pg-activity` pypi package.
   1. [x] The invocation script to automatically use credentials from `.pgpass`.
- [ ] Repeat the above steps for both the `gstg` and the `gprd` environment-specific roles.

Story points: 2.0
Praefect cloud_sql Database in staging is almost constantly near 100% CPU

For example: https://gitlab.slack.com/archives/C017L2ZV4KE/p1596693091048000

![image](/uploads/601fab9e3b845e900939f96c547107be/image.png)

https://dashboards.gitlab.net/d/alerts-sat_cloudsql_cpu/alerts-cloudsql_cpu-saturation-detail?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg&var-type=monitoring&var-stage=main&from=now-24h&to=now

Note: the service will change after https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2607, so the chart variables in the link above will need to change too. 

cc @ahmadsherif @alejandro

Story points: 3.0
Add the database user management tooling scripts to the patroni fleet

Add the database user management tooling scripts to the patroni fleet.

- https://gitlab.com/gitlab-com/runbooks/-/blob/nnelson/add-postgresql-database-user-role-utility-scripts-gl-infra-production-1847/scripts/database-gitlab-superuser-session-connection-terminate.sh
- https://gitlab.com/gitlab-com/runbooks/-/blob/nnelson/add-postgresql-database-user-role-utility-scripts-gl-infra-production-1847/scripts/database-gitlab-superuser-user-role-create.sh
- https://gitlab.com/gitlab-com/runbooks/-/blob/nnelson/add-postgresql-database-user-role-utility-scripts-gl-infra-production-1847/scripts/database-gitlab-superuser-user-role-password-update.sh

Story points: 2.0
Create new gitaly storage shard node to replace `nfs-file49`

Gitaly storage shard `nfs-file49` (`file-49-stor-gprd.c.gitlab-production.internal`) is at `60.02%` usage as of `2020-08-03`.

Update: `64.16%` as of `2020-08-12 1940 utc`

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file49`.

There are currently 7 gitaly shard nodes configured to accept new projects (`nfs-file48-54`).  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent the acceleration of capacity consumption, a new gitaly shard node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 2.0
Create new gitaly storage shard node to replace `nfs-file48`

Gitaly storage shard `nfs-file48` (`file-48-stor-gprd.c.gitlab-production.internal`) is at `62.63%` usage as of `2020-08-03`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file48`.

There are currently 7 gitaly shard nodes configured to accept new projects (`nfs-file48-54`).  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent the acceleration of capacity consumption, a new gitaly shard node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 2.0
Proxy  *.about.gitlab-review.app via Fastly

Currently about.gitlab.com is proxied via Fastly but its review apps, hosted on about.gitlab-review.app, are served directly from the `about-src` server. This means that Fastly-dependent features, like edge redirects, can't be tested currently on the review apps domain, requiring us to use a separate domain (about.staging.gitlab.com). Proxying the review apps domain would allow us to simplify our www-gitlab-com infrastructure.

Story points: 1.0
Delete provisioned VM for gl-infra-10957

A VM was provisioned for customer migration.

https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10957

Delete it in a month.

https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/ps-congregate-10957?project=transient-imports&organizationId=769164969568

Story points: 1.0
Use `general-public-splashscreen` as the default start page on dashboards.GitLab.com

I'd like to switch the default dashboard on our public grafana instance over to this:

https://dashboards.gitlab.com/d/general-public-splashscreen/general-gitlab-dashboards?orgId=1

![image](/uploads/812ecc3ff84854d3222f4827f496da81/image.png)

Why:

1. Key ARES (apdex, request-per-second, error rate and saturation) metrics for our key public-facing services.
1. Service status descriptions for each service (healthy, warning, degraded). 
1. Maintained in Git

@bjk-gitlab wdyt?

Story points: 1.0
Demonstrate runbook to add PgBouncer instances

Demonstrate https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/pgbouncer/pgbouncer-add-instance.md

- Creating a PgBouncer Read-Write node:
  - https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1925
  - https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3882
- Creating a PgBouncer Read-Only instance:
  - https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3883

Story points: 2.0
Create #elasticsearch-lab Slack bot for the Elasticsearch API

The idea is to replicate the success of the `#database-lab` Slack channel but instead for the Elasticsearch index instead of the database.

### What would it look like

There would be a new app in Slack that has read-only access to our Elasticsearch index (`grpd`) and would profile/validate the query sent to it, redacting any matching documents.

The bot would then issue the query, and upload the redacted response to the channel as a file upload. Some metadata (like the **query time**, the **result count**, etc…) could be shown directly in the bot's response. 

### Why would this be useful

As the usage of the Advanced Search ramps up, we will have to diagnose more slow queries and test improvements, such as https://gitlab.com/gitlab-org/gitlab/-/issues/225998

### Implementation

  - [ ] Do a security review of the Elasticsearch Profile API to ensure the data safety
  - [ ] Create a Slack application to manage the Elasticsearch calls
  - [ ] Add the application to our Slack workplace 

### References

  - Elasticsearch Validate API: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-validate.html
  - Elasticsearch Explain API: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-explain.html
  - Elasticsearch Profile API: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html
  - Slack's File API: https://api.slack.com/messaging/files/setup)

Story points: 1.0
Revert MR - Remove postgresql-9.6-repack extension package from the production patroni fleet

[This MR to `remove postgresql-9.6-repack extension package from the production patroni fleet`](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3866) should be reverted, that way, the `remove_package` directive which has become useless post-removal, will no longer be unnecessarily codified going forward.

Story points: 1.0
Revert MR - Remove postgresql-9.6-repack extension package on the staging patroni fleet

[This MR to `remove postgresql-9.6-repack extension package from the staging patroni fleet`](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3869) should be reverted, that way, the `remove_package` directive which has become useless post-removal, will no longer be unnecessarily codified going forward.

Story points: 1.0
Set up a GCS bucket for serving `gitlab-docs` Review Apps

Similarly to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8203#note_243150421, we'd like to host documentations Review Apps in a GCS bucket served through Cloudflare.

The setup should almost the same as for `www-gitlab-com` Review Apps.

- The domain would be `*.docs.gitlab-review.app` and managed in https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/blob/master/environments/dns/gitlab_review_app.auto.tfvars.json#L59-66.
- The GCS bucket should probably be under **Marketing > Documentation** (similarly to https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8203#note_246351579).
- The CI config for https://gitlab.com/gitlab-org/gitlab-docs should be updated similarly to https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/35869/diffs.

Related to https://gitlab.com/gitlab-org/gitlab-docs/-/issues/735.

/cc @kwiebers @axil 

/cc @alejandro since you worked on https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8203 and https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/35869.

Story points: 2.0
Monitor CloudSQL performance

As https://gitlab.com/gitlab-org/gitlab/-/issues/227215  (see also https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2383) showed, we probably need to monitor CloudSQL performance, at least for the praefect DB but possibly others as well.  Extended CPU usage above some quite high limit (90%?) is likely a sign that Something Is Wrong and needs attention, and we would have saved a lot of investigation time and delays if we'd had this as a metric that was alerting earlier.

/cc @alejandro this might be something you want to take a look at?

Story points: 3.0
Create new gitaly storage shard node `file-54-stor-gprd` to replace `file-47-stor-gprd` in the configured rotation for storing new projects

Gitaly storage shard `nfs-file47` (`file-47-stor-gprd.c.gitlab-production.internal`) is at `89.53%` usage as of `2020-07-10`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file47`.

There are currently 7 gitaly shard nodes configured to accept new projects.  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent usage acceleration, a new gitaly node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 1.0
Implement locking mechanism for database backups

We want to [run WAL-G backup push only from one of the Patroni replicas](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9627). But manually designating a node for running backups is problematic:

* It needs to be manually adjusted again in case of a failover
* Designating a "special" node isn't elegant
* Backups would stop working if the "special" node goes down

Proposal:

The replicas regularly try to acquire a lock (with a TTL) in consul and only the lock holder is running backup-push.
This lock also is used to run wal-push from a replica in the transition period from wal-e to wal-g.

* a cron-job on each replica tries to acquire the lock every minute
* the lock holder creates a local lock file (so other processes, like `archive_command`, don't need to query consul several times per second)
* the lock cronjob and backup-push script also check if they run on the primary and release the lock/lock-file in that case
* there is a 1 minute time window for race conditions in case of failover, where wal-push could run from 2 nodes in parallel for a minute, but WAL-G should have no issue with that (should test though)

Story points: 8.0
Decommission `ops-gitlab-net` ES cluster

~~The ops-gitlab-net cluster is the only one which is still on 6.x . 6.x is EOL so we should upgrade it to 7.x~~

Search indexing on ops.gitlab.net is not used. We've disabled the feature and can safely reomve the `ops-gitlab-net` deployment from our Elastic Cloud account.

Story points: 2.0
Investigate initial DNS settings on nodes in GCP and eliminate gaps in resolv.conf settings.

[Reference Incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2327)

One hypothesis from the incident was that pgbouncer will time-out cached zone records, but it won't re-read the DNS information on what upstream provider to use for resolution. This would mean that if a GCP nodes starts and the resolv.conf specifies (for example) the default Google DNS information from DHCP until our dnsmasq information in in place, pgbouncer might keep trying to query the Google DNS instead of the local dnsmasq service.

Acceptance Criterea:
1. Is this hypothesis correct?
1. If so, can we prevent the DNS gap from occuring?

Story points: 2.0
WAL-G backup failing in ops

WalG backups have been failing since May 21.

See [Thanos data](https://thanos-query.ops.gitlab.net/graph?g0.range_input=5w&g0.end_input=2020-06-24%2014%3A00&g0.max_source_resolution=auto&g0.expr=gitlab_com%3Alast_walg_successful_basebackup_age_in_hours%20%3E%3D%2048&g0.tab=0)

Story points: 5.0
Dial down our `global_search` sidekiq fleet concurrency

A request has been made to dial down our `global_search` sidekiq fleet concurrency because our elasticsearch (advanced global search) cluster is overloaded.

https://gitlab.slack.com/archives/C101F3796/p1592846811423200

![Screenshot_2020-06-22_at_20.28.03](/uploads/7ba9ce9c14849ca95d5a23d6430c51e6/Screenshot_2020-06-22_at_20.28.03.png)

Story points: 2.0
CA for Redis cluster gitlab is missing instances in ops environment

This issue tracks work to make corrective actions for "`Redis cluster gitlab is missing instances in ops environment`" alert.

It was recommended to remove the alert entirely, since there is already an alert definition for an existing instance going down. (https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2405#note_364993343)

Incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2293

MR to remove alert rule: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2405

Story points: 1.0
Enable access logging for gitlab-runner-custom-fargate-downloads S3 bucket

The Runner group would love to be able to gauge how much interest there is in our new AWS Fargate driver. The simplest way to do so would be to track downloads from the S3 bucket at https://gitlab-runner-custom-fargate-downloads.s3.amazonaws.com/master/index.html

This issue is to request enabling access request logging so that we can then parse the data to see how many downloads of the driver there are.

FYI @gitlab-com/gl-infra/managers

CC @DarrenEastman

Story points: 5.0
License Database Extract

Hi @Finotto & @gerardo.herzig

Created this issue to request a license DB extract as from the [hand book](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/#license-db)

Command required below:

`pg_dump -Fp --no-owner --no-acl license_gitlab_com_production | sed -E 's/(DROP|CREATE|COMMENT ON) EXTENSION/-- \1 EXTENSION/g' > S{DUMPFILE}`

Thanks so much again :thumbsup:

Story points: 1.0