giseldo/neodataset huggingface dataset:
Update the deleted-project-restore runbook

While doing a project restore for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10092 I encountered many issues that need to be updated in the according runbook. Basically, in most cases project restores will not work anymore as described in the runbook for multiple reasons. Need to add my notes to the [runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/deleted-project-restore.md).

Story points: 3.0
Can't use a symbolic link to bin/tf

Having establish a symbolic link to `bin/tf` from within a directory already in ones local env `PATH` in order to support easy inclusion of `bin/tf` in one's `PATH` without having to explicitly add this project's path to one's `PATH`, the `bin/tf` script fails, since its code references the actual directory of the link, not the link target, during execution.

Story points: 1.0
Add Zendesk as an external service to Status.io

Our testing environment on Status.io currently has Zendesk set up as an [external service](https://kb.status.io/monitoring/external-service/). For the sake of parity, I think we should add this to the production status page as well in the event that there's a Zendesk (and therefore, support) outage we need to report on.

Looks like admin privileges are required to add this so I'm unable to. Would you mind taking care of this, @dawsmith?

Story points: 1.0
Request for new subdomain - `advisories.gitlab.com`

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

This issue https://gitlab.com/gitlab-org/secure/vulnerability-research/advisories/landing-page/-/issues/1 requires a new subdomain, `advisories.gitlab.com`, to be created for the advisory landing pages. The project at `gitlab.com/advisories/advisories.gitlab.io` will then use the new subdomain as its custom domain for its pages.

**Details** 
 - Point of contact for this request: [+ @d0c-s4vage +]
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**SRE Support Needed**

[+ Support Request Details +]





<!--
please do not edit the below
-->

Story points: 2.0
Generate alert manager config from runbook pipeline

Currently, we have three different source of truth for our alert manager routing configuration.

These are:
1. https://gitlab.com/gitlab-com/runbooks/-/blob/master/alertmanager/alertmanager.yml.erb 
1. https://gitlab.com/gitlab-cookbooks/gitlab-alertmanager/blob/master/templates/default/alertmanager.yml.erb 
1. An encrypted file, manually stored in GCS, and not regularly updated. See https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/monitoring/-/blob/master/bin/k-ctl#L57 for details

The first two are now relatively in check with one another, but effort needs to be made to unify all three routing configs. 

## Proposal

Add a pipeline to the runbooks repo to generate the config and store it encrypted in GCS. 

Then adapt the chef repo to use this GCS file, rather than the ERB directly. 

(Placeholder, more to follow) 

------------------------------------------------------

Related:

* https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/214
* https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7071
* https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9572
* https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/16
* https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10045

cc @ggillies @skarbek @jarv @marin @craigf

Story points: 3.0
Provision new HDD-based Gitaly node in production

**Details** 
 - Point of contact for this request: [+ @glopezfernandez +]
 - If a call is needed, what is the proposed date and time of the call: no call necessary
 - Additional call details (format, type of call): n/a

**SRE Support Needed**

As part of migrating archived repositories to HDD-based storage, we need to provision a new node in production backed by HDDs. This is simiar to #9846, but in order to safeguard production, let's provision the same class of machine and adjust as we observe the behavior of the system.

<!--
please do not edit the below
-->

Story points: 2.0
investigate charts on the monitoring dashboard missing data

follow up on: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9877#note_334672627

some charts on this dashboard: https://dashboards.gitlab.net/d/monitoring-main/monitoring-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-sigma=2  show no data errors

Story points: 2.0
`ElasticCloud Watcher: gitaly_abuse_1` triggered with empty project names

The alert is triggered and posts to Slack, but is no longer including a project name:

```
Project:
File Server: file-13-stor-gprd.c.gitlab-production.internal
Average Gitaly Wall time: 2376ms/second
Average rate: invocations per second 170ops/sec
```

Example slack message (GitLab internal only): https://gitlab.slack.com/archives/CCR9GMMR7/p1588255198064400

@craigf (currently the SRE on-call) for help directing this to right place.

cc @gitlab-com/gl-security/abuse-team

Story points: 1.0
Setup Grafana Image Renderer service in k8s

Grafana recommends moving away from PhantomJS and adopting the [Grafana Image Renderer](https://grafana.com/docs/grafana/latest/administration/image_rendering/) plugin.

## Definition of Done

- [ ] The [image renderer](https://hub.docker.com/r/grafana/grafana-image-renderer) is deployed to our GKE instance.
- [ ] Slackline is verified as working with the new image rendering.

Story points: 5.0
document process of import pgbouncer logs and analysis with counts

@gerardo.herzig  I do like your approach on the incident yesterday and I would like to take notes on your steps how you proceed to :

- collect the logs from pgbouncer.
- what table structure you created 
- how did you import the data
- and what queries we could execute for analysis.

Please let me know what else can we add to the process.
Perhaps enable connection logging on Pgbouncer?

I would like to implement this and in future automate to extract and support us on the analysis of metrics from pgbouncer.

Also, we could break down the like that summarize the number of connections and the status of the pools : 

```
2020-04-29_14:03:05   2020-04-29 14:03:05.540 UTC [1136] LOG stats: 3229 xacts/s, 3879 queries/s, in 2071807 B/s, out 6585743 B/s, xact 7366 us, query 1867 us, wait 921749 us
2020-04-29_14:04:05   2020-04-29 14:04:05.540 UTC [1136] LOG stats: 3321 xacts/s, 3961 queries/s, in 2122757 B/s, out 6471056 B/s, xact 8924 us, query 3311 us, wait 6245925 us
2020-04-29_14:05:05   2020-04-29 14:05:05.541 UTC [1136] LOG stats: 3264 xacts/s, 3899 queries/s, in 2050998 B/s, out 7198025 B/s, xact 6805 us, query 2118 us, wait 2080644 us
2020-04-29_14:06:05   2020-04-29 14:06:05.541 UTC [1136] LOG stats: 4712 xacts/s, 6235 queries/s, in 3057949 B/s, out 9155166 B/s, xact 14221 us, query 7623 us, wait 38534949 us
2020-04-29_14:07:05   2020-04-29 14:07:05.541 UTC [1136] LOG stats: 4162 xacts/s, 4907 queries/s, in 2601754 B/s, out 7390354 B/s, xact 8209 us, query 3593 us, wait 9232004 us
2020-04-29_14:08:05   2020-04-29 14:08:05.541 UTC [1136] LOG stats: 4016 xacts/s, 5537 queries/s, in 2496438 B/s, out 9656314 B/s, xact 14397 us, query 7119 us, wait 32114727 us
2020-04-29_14:09:05   2020-04-29 14:09:05.541 UTC [1136] LOG stats: 3612 xacts/s, 4460 queries/s, in 2243004 B/s, out 7149377 B/s, xact 10388 us, query 4600 us, wait 12898834 us

```

And what retention time we could have in this "database"? 
We should partition the tables.

Story points: 4.0
Use `shard` labels to distinguish `pgbouncer-sidekiq` from other `pgbouncer` nodes.

Currently, it's not possible to distinguish `pgbouncer` main pool metrics from `pgbouncer` `sidekiq` pool nodes, without hardcoding database connection names or using regular expression matching on host names.

We should use the `shard` label to distinguish these metrics. Possibly `shard="sidekiq"` for the `pgbouncer-sidekiq-*` nodes. 

![image](/uploads/130d2f6799f97453b23fe75b349d14bd/image.png)

https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=count(up%7Bfqdn%3D~%22pgbouncer.*%22%2C%20env%3D%22gprd%22%7D)%20by%20(fqdn%2C%20shard%2C%20stage%2C%20environment%2C%20tier)&g0.tab=1

Minor corrective action for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9966

cc @ahmadsherif @bjk-gitlab @Finotto

Story points: 1.0
Configure chef-client in gitlab-server cookbook by default

We currently configure chef-client in our roles files. This leads to copy-n-paste errors[0].

* [x]  Add a configuration recipe to gitlab-server.
* [x]  Rollout new recipe to `gitlab-server::default`.

[0]: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/3245

Story points: 1.0
Geo replication is broken in staging after Postgres update attempt

After our Postgres update attempt on 2020-04-24, [the geo team reported replication was broken](https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/issues/12#note_53253) on staging. Although this doesn't affect the prospect of a production attempt (since geo isn't enabled in production) we should fix it to unblock the geo team.

/cc @Finotto @dbalexandre

Story points: 2.0
Reenable the usage ping in gitlab admin interface

## Overview

Reenable the usage ping in gitlab admin interface

- We had disabled the usage ping as part of the greater fixing, https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9385 .  We reenable it agains


## Next Steps

* [x] Go to GitLab.com **Admin area > Settings > Metrics and profiling > Usage statistics** and Check usage ping. 
* [x] Please post a screenshot of the usage ping settings once this once this is done.

Story points: 1.0
License Database Extract

Hi @Finotto & @gerardo.herzig

Created this issue to request a license DB extract as from the [hand book](https://about.gitlab.com/handbook/business-ops/data-team/data-infrastructure/#license-db)

Command required below:

`pg_dump -Fp --no-owner --no-acl license_gitlab_com_production | sed -E 's/(DROP|CREATE|COMMENT ON) EXTENSION/-- \1 EXTENSION/g' > S{DUMPFILE}`

Can share the files with me on Slack this time

Thanks!

Story points: 1.0
Set GITLAB_QA_FORMLESS_LOGIN_TOKEN variable on /etc/gitlab/gitlab.rb on live environments

We're getting close to having https://gitlab.com/gitlab-org/gitlab/-/merge_requests/27788 merged. With that, to allow for running tests using the formless login mechanism in live environments such as staging, pre-prod, canary, and production, we will need to set the `GITLAB_QA_FORMLESS_LOGIN_TOKEN` environment variable on `/etc/gitlab/gitlab.rb` on those environments.

You can find the variable's value at the Team 1Password under GitLab QA - Access tokens. 

Could someone help with this, please?

Cc @gitlab-com/gl-infra/managers.

Story points: 1.0
Reduce disk space contention on CI runner VMs created by gsrm

As a corrective action for https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1976, reduce the number of jobs serially handled by each runner VM from 30 to 10.

Each time a runner VM runs a job, it accumulates residual disk space used by that docker container's volumes.  Sometimes an unlucky VM runs a combination of jobs that fills its disk.  This has started happening fairly often, and it is starting to affect the efficiency of development work.

For more background, see summary notes here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1976#note_329467200

This is the cheapest of the easy options.  If it turns out to be insufficient, we can revisit the other options.

Story points: 1.0
Upgrade Patroni in gitlab.com

The goal is to upgrade Patroni to 2.0.2 without any downtime or failover.

Release notes: https://github.com/zalando/patroni/blob/master/docs/releases.rst#version-201

The initial steps that we are planning are : 

- ~~stop chef-client in all the database nodes - to do not execute any failover~~
- ~~Pause the Patroni Cluster~~
- ~~ensure that all nodes ack the pause~~
- ~~install the new version ( chef MR I guess)~~
- ~~Restart the Patroni processes~~
- ~~check that the new process is launched and operative~~
- ~~when the check is positive we resume the cluster~~
- ~~verify logs and the database and patroni are running~~

New steps (from [this comment](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9885#note_480398240)) [as of `2020-01-11`]:

* Disable chef in all the nodes.
* Update the patroni python package ( `pip` )
* Pause the patroni cluster in maintenance mode `gitlab-patronictl pause`
* Restart the patroni cluster in maintenance mode `gitlab-patronictl restart`
* Resume the maintenance mode Patroni cluster `gitlab-patronictl resume`
* Merge in chef the new version of Patroni.
* restore chef in each node.


**Acceptance criteria:**

- [x] Create a runbook to document all the processes executed.
- [x] Rollout in staging the new Patroni version.
- [ ] Execute several failovers and verify that all the integrations are working properly. E.g.: ( traffic routing).
- [ ] Enable checksums in staging using the pause strategy.
- [x] Rollout the new Patroni version in Production.

Story points: 8.0
Enable postgres_queue_enabled for Praefect in gstg/gprd

This new setting allows replication jobs to be stored on Postgresql. It isn't expected to have any impact on the normal operation of Praefect outside of replication.

See https://gitlab.com/gitlab-org/gitaly/-/merge_requests/1989 and https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4096 for more details.

Story points: 1.0
a small number of rails logs is being rejected by the ES logging cluster

## Summary
a small number of rails logs is being rejected by the ES logging cluster

## Timeline

All times UTC.

2020-04-22

- 09:17 - Incident declared from Slack

## Details
This is a continuation of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8931

Two alerts are firing for this:
- number of pubsubbeat warnings
- ES write rejections

These alerts have been firing for a few weeks now. We either need to fix the underlying problem or adjust the thresholds

## Source 
Incident declared by mwasilewski in Slack via `/incident declare` command.

## Resources
1. If the **Situation Zoom room** was utilised, recording will be automatically uploaded to [Incident room Google Drive folder](https://drive.google.com/drive/folders/1wtGTU10-sybbCv1LiHIj2AFEbxizlcks) (private)
Thanos has few alerting rules configured

https://github.com/thanos-io/thanos/blob/master/examples/alerts/alerts.md#Ruler has 43 suggested alerts for monitoring a Thanos setup.

At present, we have 4 alerts configured for our cluster.

We should consider adding the others.

For example, [one of the suggested alerts](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1d&g0.max_source_resolution=0s&g0.expr=clamp_max(%0A%20%20%20%20time()%20-%20%20max%20by%20(job%2C%20rule_group)%20(prometheus_rule_group_last_evaluation_timestamp_seconds%7Bjob%3D~%22thanos.*%22%7D)%0A%20%20%20%20%3E%0A%20%20%20%2010%20*%20max%20by%20(job%2C%20rule_group)%20(prometheus_rule_group_interval_seconds%7Bjob%3D~%22thanos.*%22%7D)%2C%201)&g0.tab=0&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=topk(5%2C%20prometheus_rule_group_last_evaluation_timestamp_seconds%7Bjob%3D~%22thanos.*%22%7D)&g1.tab=1) would point to frequent rule group failures on Thanos Ruler nodes. 

cc @bjk-gitlab @AnthonySandoval

Story points: 2.0
Investigate dips in redis-cache latency apdex, part 2

We previously investigated apdex spikiness as part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9420.

During that investigation we found apdex dips to correlate with CPU saturation. The CPU saturation has been addressed in two ways:

* Upgraded to C2 instance type https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1871
* Applied rate-limit for user producing traffic bursts https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1906

This dropped the baseline CPU utilization from 80% to 35% and it is now much smoother. That class of apdex dips no longer occurs.

However, it appears that we have a new class of latency apdex dips that does not correlate with any CPU saturation.

From the [redis-cache overview dashboard](https://dashboards.gitlab.net/d/redis-cache-main/redis-cache-overview):

![Screenshot_2020-04-20_at_20.06.21](/uploads/e55b9c4fb1dedd80dc5c19fd5c777ab2/Screenshot_2020-04-20_at_20.06.21.png)

These events are less frequent than they were previously, but the dips have a higher amplitude, which gets amplified when looking at apdex over a longer period of time.

Here is 30 days:

![Screenshot_2020-04-20_at_20.07.19](/uploads/778fb99e275a67b82f9dbfcf70222092/Screenshot_2020-04-20_at_20.07.19.png)

![Screenshot_2020-04-20_at_20.07.30](/uploads/d49af10a1432d8546ae78696c3736c8e/Screenshot_2020-04-20_at_20.07.30.png)

Please note that this dashboard uses a `min_over_time` aggregation that amplifies lower dips -- it makes things look worse than they actually are when looking at a longer time frame.

The change in behaviour appears to correlate with the exact date that we upgraded to C2 instances, March 31 (2020-03-31).

This change included:

* Change in underlying instance type from `n1-highmem-16` to `c2-standard-30`
* Change in number of CPUs from `16` to `30`
* Change in kernel version from `4.15.0-1036-gcp` to `4.15.0-1058-gcp`

It is not yet proven that this is a server-side issue. Since apdex is measured on the client-side (`gitlab_cache_operation_duration_seconds_bucket`), it is also possible that something is happening on the client.

Some possible next steps:

* Gather a CPU profile of the redis process with perf to validate the claim that there is no CPU burst during the event.
* Gather data on other processes running on the redis host during the time of the event, to validate that no other process is contributing.
* Undo some of the variables that changed, starting with the instance type: change one of the redis hosts back to a `c2-standard-30`.

Story points: 1.0
Rename the `alerts` Slack channel to `feed_alerts`

as per: https://gitlab.com/gitlab-com/support/support-team-meta/-/issues/2244 , the alerts channel needs to be renamed

this can be done by adjusting the alertmanager config, the actual names are configured in the default attributes for the alertmanager cookbook: https://gitlab.com/gitlab-cookbooks/gitlab-alertmanager/-/blob/master/attributes/default.rb#L26-33 and are partially overwritten in the chef role: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/blob/master/roles/gprd-alertmanager.json#L22-28

- [ ] try using channel IDs instead of names in AM config

Story points: 1.0
certificate for log.gitlab.net expires on 11.05.2020

SSLMate Certificate for log.gitlab.net Expiring in 27 Days

(see related email for more details)

Recently expired:

* [x] `next.staging.gitlab.com` (2020-02-22)
* [x] `monkey.gitlab.net` (2020-02-27)
* [x] `*.pre.gitlab.net` (2020-03-05)

Upcoming expiries:

* [x] `staging.gitlab.com` (2020-05-01)
* [x] ~~`*.githost.io` (2020-05-03)~~ <- set not to auto-renew
* [x] `*.testbed.gitlab.net` (2020-05-03)
* [x] `log.gitlab.net` (2020-05-11)
* [x] `*.gprd.gitlab.net` (2020-05-14)
* [x] `*.gstg.gitlab.net` (2020-05-14)
* [x] `performance-lb.gitlab.net` (2020-05-17)
* [x] `dashboards.gitlab.net` (2020-05-24)

cc @AnthonySandoval @igorwwwwwwwwwwwwwwwwwwww

Story points: 2.0
Install postgres debug symbols on hosts running postgres

To aid in profiling postgres stacks, add the debug symbol package along with the main postgres package.  This should apply to our patroni-managed hosts as well as other hosts running the `postgresql-<version>` package from the postgres apt repo.  The postgres binary is stripped of most symbols; adding the `postgresql-<version>-dbg` package will make profilers like `perf` more useful.

```shell
$ apt-cache search 'postgresql-9.6.*dbg'
postgresql-9.6-dbg - debug symbols for postgresql-9.6
...
```

Story points: 1.0
Create new gitaly storage shard node to replace `nfs-file45`

Gitaly storage shard `nfs-file45` (`file-45-stor-gprd.c.gitlab-production.internal`) is at `66.42%` usage as of `2020-04-17`.

At `78.22%` as of `2020-05-07`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file45`.

There are currently 7 gitaly shard nodes configured to accept new projects.  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent usage acceleration, a new gitaly node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 4.0
Create new gitaly storage shard node to replace `nfs-file42`

Gitaly storage shard `nfs-file42` (`file-42-stor-gprd.c.gitlab-production.internal`) is at `65.16%` usage as of `2020-04-17`.

At `77.22%` as of `2020-05-07`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file42`.

There are currently 7 gitaly shard nodes configured to accept new projects.  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent usage acceleration, a new gitaly node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 4.0
Create new gitaly storage shard node to replace `nfs-file41`

Gitaly storage shard `nfs-file41` (`file-41-stor-gprd.c.gitlab-production.internal`) is at `67.30%` usage as of 2020-04-17 and `73.51%` as of 2020-04-28.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file41`.

There are currently 7 gitaly shard nodes configured to accept new projects.  Maintaining at least this level of availability is important to avoid any shards filling up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent usage acceleration, a new gitaly node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 4.0