giseldo/neodataset huggingface dataset:
Review the latest switchover experience, consider possible improvements for future

@Finotto and @hphilipps mentioned that switchover and restart of all replicas performed (successfully) last weekend (started at 23:12 UTC 2020-06-13) caused increased load during ~30 minutes.

https://dashboards.gitlab.net/d/patroni-main/patroni-overview?panelId=28&fullscreen&orgId=1&from=1592087756529&to=1592093870453&var-PROMETHEUS_DS=Global&var-environment=gprd&var-sigma=2

![Screen_Shot_2020-06-16_at_00.08.22](/uploads/0e44980eb93d329ba083e58e563579a7/Screen_Shot_2020-06-16_at_00.08.22.png)

https://dashboards.gitlab.net/d/patroni-main/patroni-overview?orgId=1&from=1592087756529&to=1592093870453&var-PROMETHEUS_DS=Global&var-environment=gprd&var-sigma=2&panelId=31&fullscreen

![Screen_Shot_2020-06-16_at_00.08.38](/uploads/4a9cdf94750ec46dc708d5958f4d5659/Screen_Shot_2020-06-16_at_00.08.38.png)


Error rates significantly increased during that high-CPU period: 

https://dashboards.gitlab.net/d/000000144/postgresql-overview?panelId=3&fullscreen&orgId=1&from=1592089200000&to=1592092799000

![Screen_Shot_2020-06-16_at_00.12.52](/uploads/d6eb2083f674f1d0d2bb79236915b7c3/Screen_Shot_2020-06-16_at_00.12.52.png)


From what I see in monitoring, it looked like significant stress to the system, so I think the additional analysis is needed, resulting to some improvements to the procedure in the future.

Story points: 3.0
Configure Flipper HTTP adapter on gitlab.com

## Summary

In terms of ~Dogfooding, we should use GitLab's [feature flag system](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) in our [development](https://docs.gitlab.com/ee/development/feature_flags/).

Epic is https://gitlab.com/groups/gitlab-org/-/epics/3367

## TODO

TBD

Story points: 4.0
Configure Flipper HTTP adapter on staging.gitlab.com

## Summary

In terms of ~Dogfooding, we should use GitLab's [feature flag system](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) in our [development](https://docs.gitlab.com/ee/development/feature_flags/).

Epic is https://gitlab.com/groups/gitlab-org/-/epics/3367

## TODO

TBD

Story points: 4.0
Upgrade Patroni to 1.5.6

The below notes were copied from [here](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7790#note_213228185):

The [Patroni release notes for 1.5.6](https://patroni.readthedocs.io/en/latest/releases.html#version-1-5-6) cite a bug fix that allows health checks to Patroni to not be blocked by Patroni's potentially slow call to `get_cluster` (which is what most often stalls when Consul agent cannot promptly talk to Consul server).

> Reduce lock time taken by dcs.get_cluster method (Alexander)
>
> Due to the lock being held DCS slowness was affecting the REST API health checks causing false positives.

This [bug fix](https://github.com/zalando/patroni/commit/680444ae13154aca6f03556cdd1280e296b549ca) alone should help avoid *some* of the cases of unnecessary Patroni failover. The Patroni agent was sometimes slow to respond to incoming health check requests because it was holding an internal lock while running a REST call to Consul agent (which can be slow). This bug fix avoids that contention most of the time by waiting to take that lock until after the REST call to Consul finishes (and also by caching the results for up to TTL seconds).

This does not address all of our concerns (e.g. slow calls to Consul can still lead to the Patroni cluster lock expiring), so we should probably still increase Patroni's DCS `ttl` setting in addition to this upgrade.

---

Be sure to review all of the release notes between our current Patroni release (1.5.0) and the target release (1.5.6).

Confirm that Patroni cluster can run safely in mixed mode, with some members using 1.5.0 and others using 1.5.6.

Run through functional testing.

Plan to deploy in `gstg` and then `gprd`.

Story points: 5.0
Increase Patroni's patience when talking with Consul

Increase Patroni's DCS settings for `retry_timeout` (currently 10 seconds) and `ttl` (currently 30 seconds).

The rationale is described here:
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7790#note_213228185

And remember that `ttl` is silently halved by Patroni before being sent to Consul.

The `ttl` should probably be several times as large as the `retry_timeout`.

Note: The `loop_wait` setting (currently 10 seconds) may be ok to leave as-is, even though we are increasing the other settings.  `loop_wait` is roughly how long Patroni waits passively for notification from Consul of a cluster state change before it starts another round of pushing its own status updates to Consul and renewing its Consul session TTL.  The configured `ttl` must be significantly larger than both `loop_wait` and `retry_timeout`.

Story points: 3.0
We've lost a few custom metric rules since the GKE migration for the Container Registry

Since the migration to GKE a few metrics no longer work as the recording rule is invalid.

Impacted metrics:
* `gitlab_component_availability:ratio{type="registry"}`
* `slo:min:gitlab_service_apdex:ratio`
* `gitlab_service_apdex:ratio`

This situation has caused a loss of visibility into some panels on this dashboard: https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1&var-type=registry  And we've triggered a few alerts that look at this data.

Utilize this issue to find new recording rules for these metrics and ensure all panels on the aforementioned dashboard are working.

Story points: 5.0
Proposal: use imaging for provisioning

There has been some discussions on the topic, and I would like to bring it more formally and take a decision. I would like to propose to *not* provision the (at least database) nodes with chef directly, but rather from pre-created images.

The main driver is *reproducibility*. Current dynamic provisioning may lead to different versions of any or potentially many software components, like kernel (this has happened already: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7630) but could also be intermediate libraries, or main components. In any case, not having a consistent, reproducible image leads to some serious problems, like:
*  Uneven performance.
*  Difficult to impossible debugging if caused by different versioning.
*  Bugs caused on one component version not happening on another one.
*  Possible security issues or data-affecting issues (PostgreSQL, for example, is prone to index corruption issue if replicating between instances running on different versions of the glibc if they have altered the collations).

Reproducible builds and reproducible images are generally considered a good pattern. Obviously this doesn't mean chef or the current provisioning system cannot be used: but just to start from a base image, install the software, and generate the final image. Actually, images may be layered, such as in:
*  Start from base Ubuntu LTS layer.
*  Add basic common utility packages -> image
*  Add specific GitLab packages / software -> image
*  Add PostgreSQL or pgbouncer or whatever software -> image

While many options exist, [Packer](https://www.packer.io/) may be used as a tool to generate the images.

Other than reproducible builds, this technique offers other advantages:
*  Make the provisioning stage less error prone (while provisioning, some external software repository may fail, leading to the provisioning to fail).
*  Faster provisioning (normally not important, buy may help when, for instance, a new replica is needed to be brought up asap).
*  Better security. Since images are immutable, they can be statically analyzed for security issues. Dynamic provisioning may install potentially different (typically newer) versions of some packages which may not have been analyzed from a security perspective.

It also comes with some drawbacks:
*  Image management. Image combination may at some point become kind of exponential. With proper scripting, using a layered approach, this should not be a big deal (only higher storage costs of the images, but this should be very minor cost factor).
*  Software packages may be updated less. But can be fixed by creating periodic issues to review and check which software packages can be updated.

cc: @Finotto

Story points: 5.0
Evaluate Odyssey as a replacement for pgbouncer(s)

In light of the recent issues related to pgbouncer's single-threaded core saturation (like in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7632), some measures have been taken to mitigate or prevent excessive load:
*  Creating more replicas (and hence distributing the load across more servers and, consequently, pgbouncers). See: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7674
*  Adding more than one pgbouncer per PostgreSQL host and expose them as separate services, so as to have effectively multi-process load balancing via DNS. See: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7651

Obviously, none are ideal, even we bought some time. Ideally, we should use a multi-threaded load balancer. Enter [Odyssey](https://github.com/yandex/odyssey). It is heavily used at high-profile sites like Yandex, and while at version `1.0rc1`, reports from the main author claim for its stability. We should test it thoroughly and evaluate if it may become a replacement for the multiple pgbouncers per host.

Proposed plan:

*  [x] Wait until GA release, right now the latest version is an RC => https://github.com/yandex/odyssey/releases/latest
*  [ ] Determine a relevant workload and mechanism to benchmark GitLab. This might become a separate issue on its own. We really need to have a way to measure performance, reproducibly, that resembles production workload (unless anything like this already exists).
*  [ ] Measure Odyssey performance and compare with saturated pgbouncer.
*  [ ] Stress-test Odyssey and look for potential crashes and/or memory leaks.
*  [ ] Create a final report summarizing the findings.

cc @Finotto

Story points: 16.0
Unset statement_timeout for gitlab-superuser

PostgreSQL configuration parameter [statement_timeout](https://postgresqlco.nf/en/doc/param/statement_timeout?version=9.6) is currently set to `15s`. While there are reasons for this setting on a global scale (mostly to avoid possibly idling connections, taking into account that it doesn't exist any PostgreSQL parameter similar to a `session_timeout`), it causes some troubles to some administrative commands. For instance, it needs to be disabled before running some long running operations like `VACUUM` or `ANALYZE`. Most importantly, it seems to be the cause of recent `pg_rewind` failures (when triggered by Patroni). See: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543

While `pg_rewind` should be patched (by PostgreSQL), as IMHO it shouldn't be subject to the `statement_timeout`, we need to work around it. I already proposed some workarounds: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543#note_204263742 Among those, I propose to implement the following one:

> `alter user "gitlab-superuser" set statement_timeout=0` or equivalent command, to disable the statement timeout for the `gitlab-superuser` user --which would not be a bad thing anyway, unless any application connects with this user, but hopefully is not the case.

The main reason is that not only will prevent `pg_rewind` from failing, but will also avoid having to unset `statement_timeout` for several other operations.

This needs to be added to the DDL code. Required actions:
* [ ]  Check that no application code is connecting to the database as `gitlab-superuser` (should not be the case, but needs to be checked).
* [ ]  Add the above DDL statement to the DDL creation process.
* [ ]  Prepare DDL change script to apply it to production.

If there's any other proposal on how to act, please comment.

@Finotto

Story points: 1.0
Make sure meltano cert upgrade happens

Your SSLMate certificate for www.meltano.com will expire on September  7, 2019.

The renewal of this certificate is awaiting approval by the administrator(s) of the following domains:

www.meltano.com: Please visit https://sslmate.com/console/orders/www.meltano.com
meltano.com: Please visit https://sslmate.com/console/orders/www.meltano.com

For more information about how to approve this certificate, visit https://sslmate.com/console/orders/www.meltano.com

cc @gitlab-com/gl-infra/cicd-and-enablement

Story points: 1.0
Update gitlab-exporter (former gitlab-monitor) and re-enable CI metric scrapes

We disabled scraping the CI metrics because the queries are real heavy. The top issue for this is https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7724.

Now https://gitlab.com/gitlab-org/gitlab-exporter/merge_requests/101 improved the queries by adding a date range filter. This requires a new index which is going to get shipped with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32546.

Once the index has found its way to production, we should
* Update `gitlab-exporter` to 5.0.1, at least on the archive DR replica
* Revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1711 to re-enable scraping.

Story points: 2.0
alert toil: HPA Unable to Scale

This particular alert fires quite often.  I think it simply needs to be tuned.  https://gitlab.com/gitlab-com/runbooks/blob/master/rules/kubernetes-hpa.yml#L25-33  The alert is relatively basic, and when a scale down operation completes successfully, we'll have hit the minimum number of Pods, which is not a bad thing.  The last time I saw this alert fire in our staging environment, was legit to fire based on the current alert rule, but undesired as it doesn't mean anything to us:

```json
{
  "type": "ScalingLimited",
  "status": "True",
  "lastTransitionTime": "2019-09-04T11:47:26Z",
  "reason": "TooFewReplicas",
  "message":"the desired replica count is increasing faster than the maximum scale rate"
}
```

Utilize this issue to figure out how we can tune the alert such that it does not fire in situations where we scaled down to the minimum number of Pods.  In this situation, it's completely fine to happen as the HPA wanted to scale down further, but don't allow it too.  We want more than 1 Pod running for the sake of redundancy.

Story points: 1.0
Read permissions for analytics for new tables

When creating new tables, we don't assign read permissions for the `analytics` user. This has led to https://gitlab.com/gitlab-data/analytics/merge_requests/1522#note_212269236.

An alternative to updating the permissions manually is to add the `analytics` user to the `gitlab` group. `gitlab` is the user owning the tables and has full read/write access. Write is not needed for analytics, but we're only using this on the archive replica (read-only) anyways.

We may want to make sure the `analytics` user cannot be used on the patroni instances (through pg_hba, for example).

cc @Finotto

Story points: 1.0
Create shared CloudSQL module for terraform

Since the Google provided CloudSQL module does't suit our use case, we'll need to implement our own module which will extend https://github.com/terraform-google-modules/terraform-google-sql-db

The primary issues are:

1. The module does not implement private IP support. We will be connecting to our instances via a private IP from either gitlab instances or apps running on GKE clusters. Adding the private network functionality to the [VPC Module](https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/vpc) makes cloudsql work, but is currently causing problems with API permissions.
1. Google's module does not support `count` - which means that it can't be enabled and disabled with a CI variable.  When building a project with a Kubernetes cluster to attach to a project or group for Auto DevOps, we want to be able to use CloudSQL for production instances, but disable it for staging and review apps.  The module will not do this and Terraform does not support conditionally including modules (https://github.com/hashicorp/terraform/issues/12906).

The new module is here: https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/cloud-sql

/cc @ahanselka @craig

Story points: 2.0
Declare gitlab-ci GCP firewall rules in terraform

Currently, the [gitlab-ci firewall rules](https://console.cloud.google.com/networking/firewalls/list?project=gitlab-ci-155816&firewallTablesize=50) are manually managed.

We should declare these in terraform so that we can managed them in a reviewable, version controlled way.

Somewhat relatedly, some chef roles declare iptables rules, e.g. https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runners-prometheus.json#L4-28. We could consider removing these as they are redundant in GCP due to the GCP firewall. Careful checking needs to be done first to ensure that any removed iptables rule has a GCP equivalent that's enforced.

cc @jarv for fact-checking

Story points: 2.0
Fix Terraform CI failure in `aws-snowplow` environment

While working through fixing the errors in !7744 (MR: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/973) `tflint` started working properly again, and surfaced the following error in the `aws-snowplow` terraform configuration:

```
$ /bin/tflint
Error: Failed to load configurations: main.tf:152,3-21: Attribute redefined; The argument "assume_role_policy" was already set at main.tf:149,3-21. Each argument may be set only once., and 6 other diagnostic(s)
```

Several `lifecycle: ignore_changes` blocks were generating errors related to attribute names (these seem like false positives)
```
$ tflint
Error: Failed to load configurations: main.tf:671,34-35: Attribute name required; Dot must be followed by attribute name., and 5 other diagnostic(s)
```

Once all the lifecycle blocks were commented to enable `tflint` run to finish, this was the final output:

```
 $ tflint
main.tf
        ERROR:138 name must be 1 characters or higher (aws_iam_role_invalid_name)
        ERROR:138 name does not match valid pattern ^[\w+=,.@-]+$ (aws_iam_role_invalid_name)
        ERROR:148 name must be 1 characters or higher (aws_iam_role_invalid_name)
        ERROR:148 name does not match valid pattern ^[\w+=,.@-]+$ (aws_iam_role_invalid_name)

Result: 4 issues  (4 errors , 0 warnings , 0 notices)
```

Story points: 1.0
Why does postgres replication lag sometimes grow significantly

The `dr` environment's postgres replica (on host `postgres-dr-archive-01-db-gprd.c.gitlab-production.internal`) recently lagged by over 1 hour, triggering a PagerDuty alert.  The Prometheus metric `replication_lag` suggests this is may be a chronic problem potentially affecting all replicas.

Why does replication fall behind?  What can we do about it?

Story points: 2.0
Alert when running out of NAT ports

After rolling out the changes described inhttps://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5071, machines will usually not have public IPs and will access the internet via Google Cloud NAT.

To determine how many IPs we'll need, we can use the following formula

`nat_ip_count = M * P / 64,512`

Where:

M = number of machines in the region (multiply by some generous number to account for future growth)
P = Minimum NAT ports per VM (defaults to 64)

There are 64512 TCP and UDP (each) ports available per NAT IP. (https://cloud.google.com/nat/docs/overview#number_of_nat_ports_and_connections for context)

Introducing Cloud NAT will have created another resource that can be saturated. Assuming constant min NAT ports per VM, we risk running out of NAT ports as the number of machines per router grows. We should alert on this so that SREs can provision extra NAT IPs in advance.

Story points: 2.0
gitlab-monitor scrapes cause replication lag on archive replica

Background:
* We disabled `gitlab-monitor` on the patroni hosts as we suspected the queries were amplifying load on the database
* I moved the heavy queries around the `/ci_builds` endpoint to the archive replica with https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1701

We now observe [unsustainable replication lag](https://prometheus-db.gprd.gitlab.net/graph?g0.range_input=2d&g0.expr=pg_replication_lag%7Benvironment%3D%22gprd%22%2Cfqdn%3D%22postgres-dr-archive-01-db-gprd.c.gitlab-production.internal%22%2Cinstance%3D%22postgres-dr-archive-01-db-gprd.c.gitlab-production.internal%3A9187%22%2Cjob%3D%22postgres%22%2Cstage%3D%22main%22%2Ctier%3D%22db%22%2Ctype%3D%22postgres-archive%22%7D&g0.tab=0) on the archive replica:

![Screenshot_from_2019-08-30_08-27-55](/uploads/89e0f102ca1fce00dfaf2af7c0f95787/Screenshot_from_2019-08-30_08-27-55.png)

A single scrape takes about 30s and this query is the top offender (27s): 
* Query: https://gitlab.com/gitlab-org/gitlab-exporter/blob/master/lib/gitlab_exporter/database/ci_builds.rb#L10
* Plan: https://explain.depesz.com/s/d1A1

There's no direct way of speeding up the query as it is expected to scan a lot of data (about 15GB of buffers per query).

This issue is to track the infra changes.

Story points: 2.0
Create a new DR Unix group to allow SSH / sudo access

We'd like a new DR Unix group to be created which is required to allow SSH / sudo access to DR specific nodes. 

Related:

* https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/12
* https://gitlab.com/groups/gitlab-org/-/epics/575

cc @devin, @dawsmith, @fzimmer, @rnienaber 

This issue was moved from: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/issues/1

Story points: 2.0
Dashboards not syncing to dashboards.gitlab.com

```
/usr/lib/ruby/2.3.0/net/http/response.rb:120:in `error!': 401 "Unauthorized" (Net::HTTPServerException)
	from /usr/lib/ruby/2.3.0/net/http/response.rb:129:in `value'
	from ./sync_grafana_dashboards:68:in `<main>'
```
This is fetching the dashboard list from the public server (itself).  Probably broken when the public dashboard server got rebuilt; I suspect we didn't create or update the API key.

Story points: 1.0
Use LetsEncrypt for docs.gitlab.com

Similarly to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7654, I think it's time we switched the docs site (https://gitlab.com/gitlab-org/gitlab-docs) to using Let's Encrypt.

I'm really hoping this will work correctly, as https://docs.gitlab.com has a lot of visitors and is very crucial to working properly :)

Story points: 1.0
Use LetsEncrypt for www.remoteonly.org

https://gitlab.com/gitlab-com/www-remoteonly-org/ is hosted on GitLab Pages;  per https://gitlab.com/gitlab-com/www-remoteonly-org/issues/50 the cert expired a few days ago and we didn't notice.  It required manual effort to find and update the cert that sslmate had autorenewed. 

Given we have LetsEncrypt (LE) support built into Pages now (beta, but presumably generally working), we should move to that as a dog-fooding exercise, and to reduce irregular (thus cognitively expensive) toil work that can easily be missed.

Story points: 1.0
Some public dashboards are broken

For example https://dashboards.gitlab.com/d/2zgM_rImz/imported-github-importer?orgId=1

Most panels do not render, with the error message "query timed out in expression evaluation". Continuously refreshing the page occasionally yields 1 or 2 panels that load.

**Update**: Work that still needs to be done:

Remove iptables firewall rules from the runner prometheus instance so that it can be scraped by gprd infra prometheus on its public network interface. The gprd prometheus IPs should be whitelisted. [This MR](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1740) got the ball rolling but I dropped it on the floor a while ago. It's imperative that the GCP firewall is carefully checked to ensure we're not opening up a security hole by removing the on-box firewall. Add rules using terraform if need be. Note that this interacts with https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8018 which will cause the IP of the gprd prometheus to change. Currently they're behind Cloud NAT.

Story points: 2.0