giseldo/neodataset huggingface dataset:
Review of deadmans snitch infrastructure

Looking at our deadmans snitch infrastructure, as part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9694#note_326029699, and wondering whether it’s working as expected?

* The dashboard referenced from the runbook seems pretty vacant: https://dashboards.gitlab.com/d/_FOpntlmz/deadman-switches?orgId=1&var-interval=15m&var-interval=30m&var-interval=6h&var-interval=1d&var-interval=1d6h&var-environment=gprd

* Deadmans snitch only seems to be integrated with Slack, not Pagerduty (this differs from the runbook)

* Most of the alerts configured in Alertmanager do not receive any data: https://gitlab.com/gitlab-com/runbooks/blob/master/rules/deadman-switch-alerts.yml#L149, although the [`SnitchHeartBeat`](https://thanos-query.ops.gitlab.net/graph?g0.range_input=12h&g0.max_source_resolution=0s&g0.expr=ALERTS%7Balertname%3D%22SnitchHeartBeat%22%7D&g0.tab=0) does appear to be firing. 
    * Should we have one snitch per alertmanager, rather than one per environment?

* `gprd alertmanager` has not been pinged in 9 months (`ops alertmanager` is active however)

My biggest concern is that, if alertmanager stops sending alerts, at present it seems we will get a slack message to #alerts and not much else - certainly no pagerduty alerts afaics. Am I missing something here?

![image](/uploads/2d425e236ebe21d3bb51214260113caa/image.png)


## Expectations

* [x] If snitches are not getting pinged, we should be alerted through pagerduty, within a few minutes.
* [x] Are the alerts in `deadman-switch-alerts.yml` redundant or actually useful?
* [x] Is https://gitlab.com/gitlab-com/runbooks/blob/master/docs/uncategorized/deadman-switches.md up to date.

cc @AnthonySandoval

Story points: 2.0
[logging] Shard failures trying to search `json.tag: rails.application` in `pubsub-rails-inf-gprd`

The content of that tag is not structured, so a text search is necessary.

There are currently a lot of failures on queries like https://log.gprd.gitlab.net/goto/df7102d612189db2fb95d894becdcdd4.

```
Type
    query_shard_exception
Reason
    failed to create query: { "bool" : { "filter" : [ { "multi_match" : { "query" : "Successful Login", "fields" : [ ], "type" : "phrase", "operator" : "OR", "slop" : 0, "prefix_length" : 0, "max_expansions" : 50, "lenient" : true, "zero_terms_query" : "NONE", "auto_generate_synonyms_phrase_query" : true, "fuzzy_transpositions" : true, "boost" : 1.0 } }, { "match_phrase" : { "json.tag.keyword" : { "query" : "rails.application", "slop" : 0, "zero_terms_query" : "NONE", "boost" : 1.0 } } }, { "range" : { "json.time" : { "from" : null, "to" : null, "include_lower" : true, "include_upper" : true, "boost" : 1.0 } } } ], "adjust_pure_negative" : true, "boost" : 1.0 } }
Index uuid
    kiWqJuLQRG-a6Rn9iBQMpQ
Index
    pubsub-rails-inf-gprd-002303
Caused by type
    illegal_argument_exception
Caused by reason
    field expansion matches too many fields, limit: 1024, got: 1579
```

This is for a full 24 hours, but it seems reasonable to be able search in more than a few hours worth of data at a time.

Can the limit be raised or some other adjustment made to improve this?

Story points: 5.0
Execute testing of PostgreSQL upgrade using a data volume similar to production

We are executing tests of the PostgreSQL upgrade with the dataset from the staging database.

It is required to execute the test with the data set from production, to evaluate the time consumed for the process.

Please consider the following steps  to execute a consistent GCP snapshot in a read-only replica :

 - connect to PostgreSQL and execute the command `select pg_start_backup();`

 - Execute the GCP snapshot.

 - In PostgreSQL execute the `select pg_stop_backup();`

We need to attach the snapshot in the following hosts : 

 * patroni-migrate-01-db-gstg.c.gitlab-staging-1.internal

 * patroni-migrate-02-db-gstg.c.gitlab-staging-1.internal

After Ongres checks and executes the setup of the environment, we should test the playbook to execute the PostgreSQL upgrade.

Story points: 2.0
Incident Practice for Support CMOC - EMEA

Basic summary

This is meant to be a simple problem to solve and a table top scenario.
First, we are testing incident response and basic group of host interactions.

Practice environment location:  Staging?

Practice - [test env for status.io](https://app.status.io/statuspage/5bedc0c2a394fc04c9ccc974)

Scenario:
service stop haproxy on all LB - current status no haproxy running (front door is closed)

Incident Start:
EOC - execute command(s) to stop LB in gstg

Start of incident handling
1. EOC use `/incident declare` in `#incident management`
2. Skipping checking pages - we'll verify those through other tests when CMOC rotation is set.

Validate:
1. [ ] ~~EOC page~~
2. [ ] ~~IMOC page~~
3. [ ] ~~CMOC page~~
4. [x] Creation of incident gdoc
5. [x] Creation of incident issue
6. [x] CMOC / IMOC can find incident issue

Once manager and cmoc join
1. IMOC/CMOC - talk through any comments to understand the issue
1. cmoc log into status.io and talk through what they would do to create incident [Link to tests status page](https://app.status.io/statuspage/5bedc0c2a394fc04c9ccc974)
1. cmoc - talk through your update to status.io

Resolution actions
1. EOC - talk through actions you would do to get load balancers restarted
1. EOC/ Manager - talk about how to escalate to engineer on call
1. Verify incident is resolved
1. cmoc - confirm resolution and talk through status.io update
1. Follow up with action items
1. Create Incident Review issue
1. How to escalate action items to infradev.

Story points: 5.0
Increase prometheus query.max-samples

Owing to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1780, may queries need to be run directly against Prometheus. Running against Thanos will result in counter reset bugs, leading to incorrect results.

However, when querying Prometheus directly, we will frequently hit the `query.max-samples` limit, which is set at 10m samples.

This means we can neither run these queries in Prometheus or Thanos. 

![image](/uploads/2bc931db288a4da8febcfd72f136248d/image.png)


## Proposal: Increase `query.max-samples` back to the default value of 50m

Currently we limit queries to roughly 150MB of sample data (assuming 16 bytes per sample). 

The Prometheus default is 50m items or 760MB per query. 

Our prometheus fleet uses about 50% of the available memory, so this seems like a reasonable change to make, at least until the Thanos reset bug is addressed

![image](/uploads/a8e0284bc3a367147250c6b594781a94/image.png)

https://dashboards.gitlab.net/d/monitoring-main/monitoring-overview?orgId=1&from=now-12h&to=now&fullscreen&panelId=53

cc @bjk-gitlab @AnthonySandoval

Story points: 1.0
Improve stackdriver exporter metrics

We currently collect a lot of stackdriver metrics.

| header | header |
| ------ | ------ |
| ci-prd | 19959 |
| gprd | 97653 |
| gstg | 28759 |
| ops | 10732 |
| pre | 6274 |
| testbed | 92 |

This leads to a number of problems
* Large storage needs on the metrics systems.
* Slow scrape times, timeouts.
* Lost scrapes.
* Lots of stackdriver API traffic.
* Duplication of metric data between stackdriver and other monitoring components.

For example, we pull the entire `compute.googleapis.com` metrics subsystem. This is 40% of the metrics and has a lot of overlap with `the node_exporter` data.

Proposed todo:
* [x]  Filter out metrics we're not using by using more specific metric filters. (ie `compute.googleapis.com/nat`)
* [ ]  Split scrape job into more granular instances.

Story points: 3.0
Create new gitaly storage shard node to replace `nfs-file44`

Gitaly storage shard `nfs-file44` (`file-44-stor-gprd.c.gitlab-production.internal`) is currently at `66.06%` usage (`68.26%` as of 2020-04-14).

A new gitaly node should be created, and added to the list of shards configured to be included in consideration for storing new project repositories.

That way, `nfs-file44` can be removed from rotation without any concern that the node's removal from the configuration will put any additional burden on the remaining nodes.

It is important to avoid acceleration of usage growth on the remaining nodes accepting new repositories.

Story points: 3.0
Enable JavaScript source fetching for Sentry projects customersgitlabcom and customersstggitlabcom

This issue is to re-enable that setting, as we can't see the source for how the errors are being triggered.

We need this setting to be enabled in both projects:

* https://sentry.gitlab.net/settings/gitlab/projects/customersstggitlabcom/
* https://sentry.gitlab.net/settings/gitlab/projects/customersgitlabcom/

Story points: 1.0
Add chef-client-is-enabled script to the chef-client-disabler

Per [discussion here](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1898#note_318576770), in addition to intercepting calls to chef-client when it's disabled, we would also like to be able to easily check whether or not it is disabled.

Add a `chef-client-is-enabled` script to the `gitlab-server::chef-client-disabler` recipe.  Expected behavior:
* If chef-client is enabled, this script will say so and exit 0.
* If chef-client is disabled, this script will exit non-zero and show when, by whom, and for what reason (i.e. same output as though chef-client had been run and intercepted).

Story points: 1.0
Fix permissions on patroni.yml since it contains credentials

The patroni.yml file is world-readable, but it contains postgres superuser credentials.

```shell
msmiley@patroni-01-db-gprd.c.gitlab-production.internal:~$ ls -l /var/opt/gitlab/patroni/patroni.yml
-rw-r--r-- 1 gitlab-psql gitlab-psql 3752 Mar 30 06:48 /var/opt/gitlab/patroni/patroni.yml
```

Presumably only the patroni daemon needs to read this file, and the file owner is the same Unix account who runs the patroni daemon:

```shell
msmiley@patroni-01-db-gprd.c.gitlab-production.internal:~$ pgrep 'patroni' | xargs -r ps -o uid,user,args
  UID USER     COMMAND
 1071 gitlab-+ /opt/patroni/bin/python /opt/patroni/bin/patroni /var/opt/gitlab/patroni/patroni.yml

msmiley@patroni-01-db-gprd.c.gitlab-production.internal:~$ id 1071
uid=1071(gitlab-psql) gid=1071(gitlab-psql) groups=1071(gitlab-psql)
```

Therefore, it should be safe to change this file to be read/write only for its owner (i.e. chmod from `0644` to `0600`).

Story points: 1.0
Chef broken because download of origin-pull-ca.pem from cloudflare is broken

Chef-client is broken on 46 gprd nodes because https://support.cloudflare.com/hc/en_us/article_attachments/360044928032/origin-pull-ca.pem can't be downloaded by chef anymore - it seems to be captcha protected, which must be new.

https://thanos-query.ops.gitlab.net/graph?g0.range_input=2h&g0.max_source_resolution=0s&g0.expr=sum(round(avg_over_time(chef_client_error%7Benv%3D%22gprd%22%7D%5B5m%5D)))&g0.tab=0

Story points: 3.0
jobs.gitlab.com cert expired unnoticed on 2020-03-28

The jobs.gitlab.com cert expired on 2020-03-28 without us noticing.

We got informed by @joernchen at 2020-03-30 11:42 UTC. (https://gitlab.slack.com/archives/CB3LSMEJV/p1585568555313400)

We replaced the cert with a new one at 12:45 UTC by updating the chef vault.

We got an expiry warning email from SSLMate on 2020-01-28.

Questions:
- Why didn't we notice? What's the process?

Story points: 3.0
Get elastic cluster operation metrics into prometheus

As Elastic Cloud seems to not provide a way to get time series data for internal cluster metrics (we only can get the current state for things like used storage from it’s API) and the limited possibilities of the Elastic monitoring cluster to get a view at the operation of the production cluster aren't really satisfying, we should consider to setup something like https://github.com/justwatchcom/elasticsearch_exporter to get Elastic cluster metrics into Prometheus.

This also would make alerting easier and more standard compared to creating watches within Elastic and we could have dashboards in Grafana.

Story points: 8.0
Add TXT record for domain verification - Drift

We are setting up Drift for SSO in Okta, and to enable this we need to perform domain verification. 

Can we add the following TXT record to gitlab.com domain -

drift-domain-verification=fa583cfff88c496bcc62651057550656a98ab3e689c314255a1a6ae848e3e56d

Further information about this request is available in https://gitlab.com/gitlab-com/marketing/marketing-operations/issues/1193

Story points: 1.0
Upgrade DR haproxy to 1.8

staging, prod and preprod are using haproxy 1.8 and Ubuntu 18.04.2 LTS for the front-end lbs. It would be nice if DR did the same so we could unify configuration, for example using the `hard-stop-after` option in the config.

This means we have an exception in https://gitlab.com/gitlab-com/runbooks/merge_requests/1776 for the DR environment, which isn't ideal.

Story points: 3.0
Followup: Remove `gcs-` prefix from review apps

After https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8203, remove the transitional `gcs-` prefix from review apps.

After doing so, existing MRs with review apps still being server from about-src's file storage will have to be rebased from the latest www-gitlab-com master.

- [x] cookbook change: https://gitlab.com/gitlab-cookbooks/cookbook-about-gitlab-com/merge_requests/78
- [x] chef-repo cookbook bump: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2443
- [x] www-gitlab-com change: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/38161

Story points: 2.0
Create a new file module for nodes served via praefect

This will allow us to differentiate between file storages accessed directly by rails and those accessed via praefect.

- [x] gitlab-com-infrastructure changes: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1279
- [x] chef-repo changes: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2421

Story points: 2.0
Grant SELECT access to the `analytics` user in gitlab_dotcom db

I'm adding a newly created table to our data team ETL and I'm looking for somebody to grant select access to the `analytics` user for all public tables in the gitlab_dotcom db.
* For context, this is my MR where I'm blocked: https://gitlab.com/gitlab-data/analytics/merge_requests/2098
* MR for last time this happened: https://gitlab.com/gitlab-data/analytics/merge_requests/1522#note_212269236

* Other relevant issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7769

Story points: 1.0
Fix sidekiq log index mapping

Since 12/30 we get many Pubsubbeat warning alerts on pubsub-duplicate-sidekiq-inf-gprd.c.gitlab-production.internal: https://prometheus.gprd.gitlab.net/graph?g0.range_input=1w&g0.expr=rate(pubsubbeat_warnings_total%5B1m%5D)%20%3E%201&g0.tab=0 - maybe related to a deployment on that day. We should fix the index mapping.

Story points: 3.0
Gitaly error-rate exceeds SLO in the Canary stage

The name of this alert: `service_cny_error_ratio_slo_out_of_bounds_upper_5m`

PagerDuty event: https://gitlab.pagerduty.com/incidents/P01BI1F

This alert self-resolved after a few minutes, but the [dashboard shows](https://dashboards.gitlab.net/d/gitaly-main/gitaly-overview?orgId=1&from=1577133346141&to=1577154946141&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=cny&var-sigma=2&fullscreen&panelId=4) the canary-stage has briefly spiked above SLO several times today.

![Screenshot_from_2019-12-23_19-38-18](/uploads/d911065abd6c5684cf972b1addd7ec64/Screenshot_from_2019-12-23_19-38-18.png)

Story points: 1.0
Gitter beta DNS change

Gitter beta environment has recently changed. We created a new ELB and ASG for our webapp servers https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8687.

We'd like to point the DNS A record for

`beta.gitter.im` to `beta-webapp-elb-1739286740.us-east-1.elb.amazonaws.com`.

(I'm not sure how that is usually done outside of one AWS account).

### Additional notes:
The `beta-webapp-elb` ELB is in Gitter's AWS account.

Right now, the `beta.gitter.im` DNS record points directly to one of our EC2 instances [gitter-beta-01](https://console.aws.amazon.com/ec2/home?region=us-east-1#Instances:search=i-007a3fd34e0535967;sort=tag:Name)`54.226.100.160`

Story points: 1.0
Remove residual firewall rule definitions that would re-apply on next reboot

Recently some iptables DROP rules were unintentionally applied to Patroni hosts, causing connectivity loss to the database servers ([incident](https://gitlab.com/gitlab-com/gl-infra/production/issues/1421) and [review](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8528)).  The immediate recovery steps included unassigning Chef role `gitlab-iptables` and manually remove iptables rules from the affected hosts (especially the Patroni hosts).

That left some residual unmanaged configuration on those hosts.  As part of an incident follow-up corrective action, that residual config was [discovered and found to probably be dangerous](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8600#note_261924370).

The "gitlab-iptables" cookbook's "default" recipe uses cookbook "iptables-ng" to define iptables rules.  That "iptables-ng" cookbook installs deb package "iptables-persistent" (and implicitly also the "netfilter-persistent" deb package), whose job is to persist those iptables rules across reboots.

The "iptables-ng" cookbook stores its version of the rules in `/etc/iptables.d/[table]/[chain]/[named_rule]`, but as far as I know, those are only used by chef-client to support that cookbook.  So those files by themselves may be harmless.

In contrast, the contents of `/etc/iptables/rules.v4` and `rules.v6` are actively loaded during reboot by the "netfilter-persistent" systemd service.

For all hosts that unintentionally had the "gitlab-iptables" cookbook applied, if that cookbook has been removed and not reapplied, we should apply some clean up steps that at least include one of the following:

* Option A: Disable systemd unit "netfilter-persistent.service"
* Option B: Remove files /etc/iptables/rules.*
* Option C: Remove deb package "iptables-persistent" and optionally also "netfilter-persistent"

Story points: 1.0
Many `PagesDomainSslRenewalWorker` sidekiq exceptions

**Queue: `pages_domain_ssl_renewal`**

Today I noticed that there have been many instances of `PagesDomainSslRenewalWorker` sidekiq exceptions lately.

Here is the grafana chart for the past 36 hours:

![Screen_Shot_2019-12-17_at_12.15.10_PM](/uploads/2c8c58290c31acb6146fa41a9bc1b118/Screen_Shot_2019-12-17_at_12.15.10_PM.png)

Here are the logs for the past 7 days, with 320,716 hits, about 45,000 errors per day.

![Screen_Shot_2019-12-17_at_12.42.37_PM](/uploads/65c261c0ddbcefa65d914e44dae94056/Screen_Shot_2019-12-17_at_12.42.37_PM.png)

The errors in the logs tend to be one of the following:

```
Acme::Client::Error::RateLimited: Rate limit for '/acme' reached

Acme::Client::Error::Timeout

Faraday::ConnectionFailed: Net::OpenTimeout

Faraday::ConnectionFailed: Connection reset by peer - SSL_connect

Acme::Client::Error::BadNonce: JWS has no anti-replay nonce

Acme::Client::Error::UnsupportedOperation: Directory at https://acme-v02.api.letsencrypt.org/directory does not include `new_account`
```

Story points: 3.0
Evaluate Elastic APM as distributed tracing solution

Elastic APM has an OpenTracing bridge and thus could be an easy plugin solution to send tracing data to (https://gitlab.com/gitlab-org/labkit/merge_requests/40). We should evaluate if we can use it for distributed tracing of GitLab.com.

* technical requirements
* are there missing features (e.g. which features of OpenTracing are supported/missing)?
* correlation with logs?
* estimate data volume
* cost factor

Story points: 5.0
Help I.W. as an onboarding buddy

As an onboarding buddy for I. W., I want to make the process for him as smooth as possible.

* [x] Introductory Coffee chat
* [x] Make sure he has access to his onboarding issue (https://gitlab.com/gitlab-com/people-group/employment/issues/1678).
  * [x] Clarify if new account still needs to be created (https://gitlab.com/gitlab-com/people-group/employment/issues/1678#note_260789516).
* [x] Provide resources
  * [x] relevant slack channels
* [x] Schedule follow-up meeting in first week of new year.
* [x] Make sure there is an SRE onboarding issue.
* [x] Help with SRE onboarding
* [ ] See if we can meet in person on a co-working day.
* [ ] Improve onboarding issues / handbook where necessary.

Story points: 5.0
Remove deprecated Digital Ocean instance of design.gitlab.com

Since `design.gitlab.com` has been moved to a Kubernetes cluster, managed by [the project's Auto Devops configuration](https://gitlab.com/gitlab-org/gitlab-services/design-gitlab-com/), it is time to remove the previous Digital Ocean instance of the application.

This Droplet is in the `GitLab Production` Project in the Digital Ocean console.

Once the instance is deleted, the chef configuration should be removed from the chef server.

Story points: 3.0
Remove deprecated AWS instance of version.gitlab.com

Since `version.gitlab.com` has been moved to a Kubernetes cluster, managed by [the project's Auto Devops configuration](https://gitlab.com/gitlab-services/version-gitlab-com/), it is time to remove the previous AWS instance of the application.

Once the instance is deleted, the chef configuration should be removed from the chef server.

The instance is in `us-east-1` and it is named `redash.gitlab.com`

Story points: 2.0
Setup logging for Praefect on gprd

Most of the work was done when setting logging on gstg (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8142) we just have to create the pubsub host.

- [x] gitlab-com-infrastructure MR: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1237/

Story points: 2.0
Cloudflare: Fix HAProxy ACLs to also check Cloudflare headers

From @asaba in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8409#note_259451396:
> The `require-recaptcha.lst` is used here: https://ops.gitlab.net/gitlab-cookbooks/gitlab-haproxy/blob/master/templates/default/haproxy-frontend.cfg.erb#L141. When configured like that, do you know if haproxy will check against additional IP address associated with the request (e.g. the extra X-Forwarded-For from cloudflare)?

We need to fix all ACLs using `src` to also look at the Cloudflare headers when `src` is a Cloudflare IP.

Story points: 5.0
Hosted discourse forum PoC testing / readiness review

Once we have a PoC environment provisioned and accessible under &139, with a recently [restored backup](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8677) and [authentication configured](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8678), we need to perform thorough validation. This should include working through a [readiness review](https://gitlab.com/gitlab-com/gl-infra/readiness).

Story points: 1.0