giseldo/neodataset huggingface dataset:
create config setup in chef of the new PostgreSQL 11.6 cluster

We need to be able to install the package, create the folder structure, and configuration files  of postgresql 11.6 on the hosts with chef. as we currently do for postgresql 9.6.

Also, we need adapt monitoring, backups and the patroni setup.
At the moment we have a recipe to make the package available.

We need to add to our chef to :

- install the package of PG 11.6 
- create the setup for the cluster ( folder structure )
- render the config file from chef ( the config from the chef to the proper patroni files )
- change some monitoring functions from for Postgresql 11.6 ( there are some changes in the name of some functions )
- Point the backups to the data folder from the Postgresql version 11.6
- change the patroni setup to work with the data folder from pg 11.6

Story points: 8.0
create chef recipe/cookbook to install extensions on postgresql 11.6

We need to make available pg_repack for the new pg cluster 

we need to install the extensions  pg_Repack and pg_Stat_Statements.

We should add this config on the new cluster to start with if not would require a new restart in the future.


It should be added on the cookbook/recipes for pg 11.6.

Story points: 4.0
create script for PostgreSQL upgrade

we need to script all the steps the database migration ( could be in ansible): 
 - consider that all the steps will be executed one-by-one and we can cancel the total execution in case of any failure
 - would execute them from a bastion in a shared session as screen
 - the steps will be : 


** Pre checks : **
- Verify files ownership
- Check Version before the upgrade	
- Verify check of pg_upgrade
- Delete and verify the monitoring functions are deleted.
- Verify the collation between versions.

** Upgrade steps : ( migration script) **
- stop all the connections and verify that
- stop the 9.6 cluster
- Take a snapshot from the database.
- Execute pg_upgrade ( all nodes )
- Apply pg configuration changes ( parallel query parameters ( parallel )  and others… autovacuum maybe ) ( all the nodes ) ( MR to executed separated )
- Start cluster 
- Vacuumdb analyze ( primary )
- Vacummdb freeze ( primary)  - it will run for longer … we will start with QA tests and enabling traffic.
Changes on PG_exporter ( all nodes )  



** Post checks : ( migration script) **
- Check version after upgrade
- Create views and functions taht we need for monitoring and verify they are created properly
- Change some functions for monitoring in exporter

Story points: 4.0
DNS and certificate update for Bouncer

I'm currently in the process of moving [Bouncer](https://gitlab.com/gitlab-com/security-tools/janitor-rails), an internal antispam tool, over to GCP so that everyone who isn't me can actually access it:

* I'd like to point the DNS entry for `bouncer.sec.gitlab.net` to 35.224.46.119 (the IP address of the GCP machine I've provisioned for this)

* And I'd need an updated HTTPS certificate due to this move (and also, in a rare instance of serendipitous timing, the old cert just expired)

Thanks, and let me know if there's anything else needed on my end!

Story points: 1.0
Design database infrastructure for Praefect

In order for Praefect to support replication coordination and failover it will rely on a PostgreSQL database. We must define a cost-effective and performant infrastructure to support this, with replication in mind.

More info on Praefect's data model: https://gitlab.com/gitlab-org/gitaly/issues/1495

- [x] https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1468
- [x] https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/2816
- [ ] Opt-in to CloudSQL maintenance notifications (to be addressed on https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9462)
- [x] Configure preferred maintenance window day of the week and time https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1480

/cc @gitlab-com/gl-infra for thoughts

Story points: 6.0
Clarify durations in deployer annotation text

Clarify durations in annotation text.

Old text printed the 2 durations adjacent to each other, resulting in
confusing phrasing such as "an hour and 16 hours":

> deployer finished a deployer pipeline of ... on gprd which took
> an hour and 16 hours (wall time)

This change aims to separate and more clearly define the two durations:

> deployer finished a deployer pipeline of ... on gprd which had 
> end-to-end wall clock duration of an hour (and sum of pipeline
> stage durations was 16 hours)

Story points: 1.0
CI job stalled with no trace output for chef-repo apply_to_prod job

Investigate why a CI job in our chef-repo's CI pipeline seems to have stalled indefinitely.

### Details

The following manually triggered job (`apply_to_prod`) in the normal chef-repo CI pipeline seems to have stalled and shows no job trace output via the web UI:

https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/jobs/909330

The purpose of this job is to push the merge request's changes to chef server, so that subsequent chef-client runs can apply them.  Chef server did get those changes, which initially suggested that this job may have run but lost its output.  However, we discovered that this job had been run twice, with the first run succeeding and the second run stalling.

https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/jobs/909204

So the successful first run would have pushed the changes to Chef Server.  That puts us back to suspecting that the 2nd job run is stalled rather than just not reporting its output.

Story points: 1.0
Postgres Replication lag is over 3 hours on archive recovery replica

Postgres Replication lag is frequently over 3 hours on archive recovery replica, and was especially high on Tuesday, 2020-02-11.

![Screen_Shot_2020-02-13_at_10.10.28_AM](/uploads/6d30a5240151878cfd71a8c549127981/Screen_Shot_2020-02-13_at_10.10.28_AM.png)

https://prometheus-db.gprd.gitlab.net/graph?g0.range_input=1w&g0.expr=(pg_replication_lag%20%3E%201)%20and%20on(instance)%20(pg_replication_is_replica%7Btype%3D%22postgres-archive%22%7D%20%3D%3D%201)&g0.tab=0

Some initial diagnosis work has been undertaken by @gerardo.herzig with `#ongres-gitlab`.

> "From time to time, wal-e wal fetching is getting stuck"

Please see the linked incident issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/issues/1653

Story points: 3.0
It appears that the pgbouncer_async_pool component of the pgbouncer service consistently exceeds SLO

This alert popped up in the #alerts-general slack channel today:

> The `pgbouncer` service (`main` stage), `pgbouncer_async_pool` component has a saturation exceeding SLO and is close to its capacity limit.

> This means that this resource is running close to capacity and is at risk of exceeding its current capacity limit.

So, I clicked through to the dashboard and saw this:

![Screen_Shot_2020-02-12_at_10.00.07_AM](/uploads/dda0e529a7c5caaa15a92061fd8c98e2/Screen_Shot_2020-02-12_at_10.00.07_AM.png)

https://dashboards.gitlab.net/d/alerts-saturation_component/alerts-saturation-component-alert?orgId=1&from=now-7d&to=now&panelId=2&tz=UTC&var-environment=gprd&var-type=pgbouncer&var-stage=main&var-component=pgbouncer_async_pool

That looks pretty consistently not good.

I imagine the pool needs to be adjusted upward, to expand capacity?  I do not yet have any knowledge about whether the existing system resources will support such an expansion or not.

Story points: 4.0
check the performance of the bloat check

We had this check offline - Get http://10.217.7.101:9168/database_bloat: context deadline exceeded

https://prometheus-db.gprd.gitlab.net/targets#job-gitlab-monitor-database-bloat

Seems we reached a timeout.

We should investigate why, and find out, performance improvements if possible.

The queries come from an external codebase recommendation:

https://gitlab.com/gitlab-org/gitlab-exporter/-/blob/master/lib/gitlab_exporter/database/bloat_btree.sql

https://gitlab.com/gitlab-org/gitlab-exporter/-/blob/master/lib/gitlab_exporter/database/bloat_table.sql

Story points: 4.0
create test environment in staging for capacity planning

considering the issue : https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9094

We need the necessary environment to proceed with the [GPT](https://gitlab.com/gitlab-org/quality/performance):

- [x] database server with the setup similar with production
- [x] create a box to deploy the performance tool.
- [ ] Configure boxes for [pointing Gitlab instance to the database box](https://docs.gitlab.com/ee/administration/high_availability/gitlab.html).

Story points: 2.0
Regenerate docker-machine certs during gitlab-runner upgrades

### Task

Opportunistically regenerate the CA and client certificates used by `docker-machine` on our CI Runner Manager hosts to provision VMs.  This avoids a race condition that can corrupt the certs if they expire naturally while the host is in service.

We deploy the `gitlab-runner` service more often than the certs expire, and that planned downtime provides an opportunity when no other `docker-machine` processes should be running, hence avoiding the race condition.

### Background

Currently our CI Runner Manager hosts use `docker-machine` to provision VMs for running CI jobs in docker containers.  The manager host acts as a CA and signs its own client cert as well as the per-VM server cert.  The CA and client certs have a 3-year lifespan, and when they expire, the next invocation of `docker-machine create` will generate a fresh pair of certs.  However, in a high-throughput environment like GitLab.com, it is very likely that multiple docker-machine processes will concurrently try to generate new certs, each one overwriting the other's work.  This can lead to mismatched or corrupt CA and client certs.  When that occurs (see https://gitlab.com/gitlab-com/gl-infra/production/issues/1609#note_283265339), the host running docker-machine can provision new VMs, but those VMs reject all attempts to authenticate to its `dockerd` daemon using the damaged or untrusted client cert.  Consequently, the manager host fails to run every job it pulls.

Story points: 1.0
Terraform error re-creating forwarding rule for load balancer

When deleting a load balancer and then re-creating it and its forwarding rule, we get an error due to not specifying the subnet.

```
Error: Error creating ForwardingRule: googleapi: Error 400: Invalid value for field 'resource.subnetwork': ''. Network interface must specify a subnet if the network resource is in custom subnet mode., invalid

  on .terraform/modules/gcp-tcp-lb-internal/loadbalancing.tf line 102, in resource "google_compute_forwarding_rule" "internal":
 102: resource "google_compute_forwarding_rule" "internal" {
```

Story points: 2.0
Evaluate options to enable checksums on the database

We need to enable checksums in the database, the main reason is to avoid data corruption.

We need to estimate the most adequated strategy to accomplish this task on our database.

We could consider a switchover to an instance generated by logical replication. (It is needed a project to create this instance from the actual cluster using logical replication.)

Or we could think about a longer maintenance window to enable checksums on the database, but our SLA agreements won't allow this approach.

Story points: 12.0
Create image for testing the PostgreSQL upgrade

I would suggest, create an image before we execute the pg_upgrade.

I would suggest stopping the PostgreSQL instances to keep the databases in a consistent state.

The suggestion would be to have a cluster ( can be with 2 nodes only Read-Write and Read-Only). 
With Postgresql 9.6 ( the actual version from Production ) and version 11.6 that is the target for this migration.

Also, we should consider having the restoration of a backup in the PG 9.6. 

With this image we could restore the testing scenario easier after our test, to be able to iterate faster.

Story points: 4.0
Capacity Planning for the primary database (closed)

It has been a long time since our last capacity planning, and we would like to verify:
- How much more load we could add on the primary database without degradation in the performance.
- We are reaching 60k connections per second. How much could we raise safely?
- Should we increase disk or memory to optimize performance? Any suggestions in shared buffers setup?
- Estimate how much % of the load is related to the table merge_requests.
- Add more topics that you think could be the next bottleneck to scale the availability of our primary database.
- measure the hot set of data that we use at the moment in memory. How much are we hitting disk. (`pg_buffercache` and the LRU count)

Story points: 12.0
Verify the optimized parameters for PG Upgrade

Our goal is to upgrade our database cluster to PostgreSQL to version 11.6, and we need to verify the optimized setup of parameters of PostgreSQL and Patroni/Consul for our database cluster. 

Please consider the parallelism parameters that we could optimize.

Add here your thoughts about new parameters or values that we could discuss better.

Also, check the optimized parameters for the pg_upgrade command that we will execute.

This task is a top priority since we need to execute this migration until April.

Story points: 8.0
Create test box to simulate postgresql upgrade

We need a staging box, with the following 2 different versions of postgresql installed :

- PG 9.6 as we have in production
- PG 11.6 that is the target of our migration

- Please create a box with the same specs as production.

For this we need :

- [ ] create the new cookbooks needed to complete the task of execute a Postgresql migration

To execute the initial test : 

Restore a backup from staging or execute and restore a dump from the database.

With this steps we will be able to execute the pg_upgrade.

Story points: 4.0
Document the TF and Chef Provisioning and Bootstraping Process in the Runbooks

## SOC Type I Control Remediation Step for VM Provisioning

The Security Compliance team is preparing for a SOC external audit, see https://gitlab.com/gitlab-com/gl-security/compliance/compliance/issues/1381. And has asked for simple documentation on the build process–provisioning and bootstraping–of servers in our production environment. We perform this workflow with Terraform and Chef, but I'm unable to locate a simple explanation on the process. Perhaps I've overlooked it, but I couldn't find anything in the runbook directory in https://gitlab.com/gitlab-com/gitlab-com-infrastructure that outlines the workflow.

## Definition of Done

In a Markdown file in the runbooks directory:

- [ ] identify which machine image gets used for VM creation, preferably by pointing to the code
- [ ] include a workflow diagram that outlines both human and machine interactions with the files and tools that provision and bootstrap the image

Story points: 1.0
Prometheus config is not being reloaded on rules change

From slack (while it lasts): https://gitlab.slack.com/archives/CK171RT0F/p1580460508118300

>>>
Some changes to the runbooks got merged to master last night but didn’t get picked up by Prometheus. On investigation, the changes has been rolled out to the `rules` directory, but it looks like chef didn’t tell Prometheus to reload its configuration. 

I hupped the process and all is now right in the world…
>>>

We should ensure that chef performs a config reload on changes.

Story points: 1.0
PagerDuty Services, Escalation Policies, and Users in Terraform

PagerDuty recently published an [article](https://www.pagerduty.com/eng/how-why-terraform) advocating for IaC with PagerDuty objects in Terraform. This is a great idea! Or is it?

This issue seeks to collect feedback on the pros and cons of doing so, before commiting to putting putting these resources into version control.

Story points: 2.0
Configure replication in praefect

For this we'll need to create a new storage node and add it to the existing virtual storage. All settings required are already available in omnibus

- [x] terraform changes: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1353
- [x] chef-repo changes: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2576

/cc @johncai

Story points: 3.0
Error check on querying the gitlab.com replica

I'm trying to debug why our extract for certain tables is failing from the gitlab.com replica. We're trying to query the `resource_label_events` table and it seems like something is killing the job. I can't tell if it's a timeout on the postgres side or something on our side. 

The latest query was between these times UTC:

```
[2020-01-29 23:08:01,384] INFO - b'INFO:root:SELECT id , action , issue_id , merge_request_id , epic_id , label_id , user_id , created_at , cached_markdown_version , reference , reference_html FROM resource_label_events\n'
[2020-01-29 23:08:01,384] INFO - b' \n'
[2020-01-29 23:14:13,961] INFO - Event: gitlab-com-db-scd-5c81df55 had an event of type Failed
```

Maybe it's related to something like https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7769 ?

@Finotto would you be able to provide some guidance / insight here?

Happy to provide whatever information I need on my end.

Story points: 2.0
Allow Rate Limiting Specific URL's

Currently, there is not a straightforward way to rate limit access to specific URL's.  We can either allow a URL or block it.  It would be very useful to be able to set limits on how many times per minute a single client can request a URL, or how many times per minute a URL can be requested in total.  A flexible way to specify this would add significantly to the site's resilience.

## Wishlist

- It would be nice to be able to specify a URL regex, for example to rate limit all `*.zip` files.
- It would be nice to be able to rate limit groups of users differently, including unauthenticated users
- It would be nice to be able to rate limit IP's and CIDR ranges separately from users.
- It would be nice to be able to limit on other criteria such as User-Agent

## Things to consider

- Some larger organizations will be behind NAT's and all users will appear to be from the same IP
- Cloudflare may appear to make a lot of requests from the same IP, so we'll need to be able to differentiate the proxy server from the end user
- A DDoS may be distributed, and may appear to come from many IP's.  We should still have a way to set a high threshold for how many times a specific URL can be requested at all.  This should be much higher than the single IP threshold
- We should be careful not to limit internal components such as sidekiq inadvertently

## Related things

- Here is an incident that this would have prevented: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9047
- Here is some related work happening in the product: https://gitlab.com/gitlab-org/gitlab/issues/30829
- Here is a related corrective action for API rate limiting: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6645

Story points: 5.0
Replace staging certificate in Cloudflare

The certificate for `staging.gitlab.com` has been updated in the GKMS vault and on the servers.  The key was updated as well. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8965

There is a copy of this cert and key in Cloudflare which is still the old version.

The new certificate is:
```console
$ openssl s_client -connect geo.staging.gitlab.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=7A:E2:52:DB:CB:40:B5:D1:E8:2F:CF:C6:74:23:DF:90:41:76:91:3F
```

The cert and key are in the `gitlab-omnibus-secrets-geo gstg` GKMS vault. They can be retrieved like this:

```
./gkms-vault-show gitlab-omnibus-secrets-geo gstg | jq -r '."omnibus-gitlab".ssl.certificate' | gsed 's/\\\n/\n/g'
```
and
```
./gkms-vault-show gitlab-omnibus-secrets-geo gstg | jq -r '."omnibus-gitlab".ssl.private_key' | gsed 's/\\\n/\n/g'
```

The Cloudflare cert and key need to be updated to the new version of the cert and key.

Story points: 1.0
Scale up praefect nodes for more CPU cores

On https://gitlab.com/gitlab-org/gitaly/issues/2348#note_275054258 it was established that the periodical CPU spikes we're seeing in praefect nodes are actually caused by chef-client runs, and it was suggested that increasing the CPU core count could mitigate this problem. Let's change from `n1-standard-1` to `n1-standard-2`.

Story points: 1.0
Split out secrets for geo secondary in staging

The Geo Secondary node in staging currently uses the same `gitlab-omnibus-secrets` file as the rest of the environment.  In some cases this is fine, but in cases like the redis server, we want to not configure secrets and use the local omnibus instance.  This doesn't work with secrets configured because Chef assumes that if we have them, we must want to use them.

The proposal (https://gitlab.com/gitlab-org/gitlab/issues/37926#note_274534850) is:

- Change the chef configuration for that node to use a different secrets file here: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gstg-infra-geo-secondary.json#L36
- Then delete secrets from that new file which we don't want applied to the node
- Cange any other secrets which shouldn't be the same (This is probably out of scope for this issue)

/cc @craigf @dbalexandre

Story points: 3.0
Configure External Access for geo.staging.gitlab.com

In order to work with the geo secondary instance, we need to be able to get to the admin interface. https://gitlab.com/gitlab-org/gitlab/issues/37926
I have tried a few different approaches to this.

- Using our existing haproxy is dangerous, since it requires major changes to the template used by all instances (including production) in order for haproxy to proxy requests for geo.staging.gitlab.com to the geo-staging instance.
- Using a GCP load balancer is a little heavy handed since we only have one backend, but I tried it.  I discovered that we would have to change the module type that we use to provision the geo-staging instance from generic-store to generic-sv-with-group - or write a whole new module. This is in addition to adding individual resources for health checks, backends, url_maps's and various other things.  Writing a whole new module is the right way to do this, but that's too much work for a one off which will only be used for development work.
- Provisioning an external IP address and creating a firewall rule to allow access directly to this node.  This is what I'm going with for the moment, since it is the simplest and there is no need for anything more complicated.  This will get us unblocked and we can revisit it in the future.

To provision the external address we need:

- A static external IP associated with the node
- A Firewall rule allowing this access to the node
- A DNS entry
- A certificate or SAN for SSL connections to the interface

Story points: 3.0
"Number of PostgreSQL Databases in gprd has changed in the past minute" alerts

The name of this alert: "Number of PostgreSQL Databases in gprd has changed in the past minute"


"Number of PostgreSQL Databases in gprd has changed in the past minute" alert was received twice during Sep 10-11.

  - The first alert (sep 10): https://gitlab.slack.com/archives/C3NBYFJ6N/p1536587684000100
  - The second one (sep 11): https://gitlab.slack.com/archives/C3NBYFJ6N/p1536678976000100

@skarbek noticed, that nothing appeared to change in the database configuration, so something might be wrong with the way the alerts are working.

Todo: check the code for this alert and how reliable it is.

/cc @gitlab\-com/gl\-infra

Story points: 3.0