giseldo/neodataset huggingface dataset:
Help create temporary access for GCP bucket for Package team working on Registry

We have https://gitlab.com/gitlab-com/team-member-epics/access-requests/-/issues/7318#note_435195251 open to help the team working on registry better understand current development work.

From the AR:
Justification for this access: Hayley and I are both maintainers of the GitLab Container Registry. As part of the upcoming registry upgrade (https://gitlab.com/gitlab-org/container-registry/-/issues/191), we will need to create an inventory of the repositories that exist in the current registry buckets. We may also need to examine these during the upgrade. For this reason, we need to be able to list/scan the buckets using the GCS API. This requires a service account with read access (`Viewer` role, I believe). Having this access is also useful to debug customer issues in general.

Decide if we want to make a service account or just give viewer role to the two developers temporarily.

Story points: 2.0
Create new gitaly storage shard node to replace `nfs-file51`

Gitaly storage shard `nfs-file51` (`file-51-stor-gprd.c.gitlab-production.internal`) is at `79.12%` usage as of `2020-10-12`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file51`.

There are currently 7 gitaly shard nodes configured to accept new projects (`nfs-file51-57`).  Maintaining at least 7 or 8 shards with sufficient capacity for new user repository creation is about the target level of availability we prefer and is important because it helps us avoid a scenario in which shards fill up too quickly.

To remove a single node from the new projects storage rotation cluster, and also prevent the acceleration of capacity consumption, a new gitaly shard node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 1.0
DNS Audit- Please review these IPs

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 
 - Point of contact for this request: @mmaneval20 
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**SRE Support Needed**
We are working with a third party security scanner, Bit Sight. They have identified the following IPs as associated with GitLab via DNS records. We have found that a few of them were given up but perhaps the DNS record wasn't updated. 

My ask is that you review [this list](https://docs.google.com/spreadsheets/d/1O6FbyB-uGpVbWjZtEAwVxeM53Db71YMcb2fLUvBUfqI/edit?usp=sharing_). If there are any IPs that are no longer ours, please indicate that on the Google Doc. Then, please validate that the DNS record was properly removed/updated. 

If they are ours, could you please indicated if it is: Production, Pre-prodcution, or User Managed. 

I will then get any IPs removed from the Bit Sight Portal. 

Please note that at this time, I'm not saying there is necessarily anything vulnerable or wrong with these IPs. However, we NEED to validate these IPs so we can remove the misattributed IPs that could be negatively impacting our score. 

If it's easier to get on a sync call, let me know. 



<!--
please do not edit the below
-->

Story points: 2.0
Automate k8s-workloads sidekiq configuration updates

Instead of creating a MR in the `k8s-workload/gitlab-com` project with an empty yaml file and then reverting the MR in order to execute a noop change to trigger a pipeline run, or from clicking a button on a pipeline web page, switch to using [the programmatic method for triggering a pipeline build](https://gitlab.com/gitlab-com/gl-infra/production/-/blob/c92bba1c33cee8d215d9822e5fa0c4a62beef054/.gitlab/issue_templates/storage_shard_creation.md#run-k8s-workload-master-branch-pipeline) and include its execution as part of a pipeline stage when changes are merged to the main branch of our chef repo project.

Story points: 1.0
Container registry - defining PostgreSQL setup for prod

We would like to consider in this issue what will be the production setup for the container registry database.


We had some chats on the issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11294

I would like to summarize here the actual idea of the setup for the PostgreSQL for production:

Hardware spec: `n-standard-32` with 64 GB memory.


Architecture:

Using the production consul cluster have an entry for our Patroni Cluster.

We will have 3 nodes initially on the cluster:
1-Primary for read-write traffic.
1-Secondary for read-only traffic.
1-Secondary that will not receive traffic to take snapshots and backups.

Have 2 pgbouncers nodes in front of the primary database node for the read-write traffic.
Have 2 pgbouncer nodes in the read-only servers.
 

Let's enable data checksums in the database from the initial setup.


Postgresql Setup:

Story points: 8.0
Create new gitaly storage shard node to replace `nfs-file50`

Gitaly storage shard `nfs-file50` (`file-50-stor-gprd.c.gitlab-production.internal`) is at `82.50%` usage as of `2020-09-29`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file50`.

There are currently 6 gitaly shard nodes configured to accept new projects (`nfs-file51-56`).  Maintaining at least 7 or 8 shards with sufficient capacity for new user repository creation is about the target level of availability we prefer and is important because it helps us avoid a scenario in which shards fill up too quickly.

Note: I have already removed `nfs-file50` from the configuration to receive new repositories, because it has become well past the 80% threshold by which it should have been removed already.

To remove a single node from the new projects storage rotation cluster, and also prevent the acceleration of capacity consumption, a new gitaly shard node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 1.0
2020-09-29: Silence alert for: The authorized_projects queue, main stage, has a queue latency outside of SLO

Silencing alert for: The authorized_projects queue, main stage, has a queue latency outside of SLO (`sidekiq_background_job_queue_apdex_ratio_burn_rate_slo_out_of_bounds`)

Duration: `30 days`

Request for silence: https://gitlab.slack.com/archives/CB3LSMEJV/p1601395587142400

Feature category: `authentication_and_authorization`

Story points: 1.0
Add zone checks to firewall rules for Cloudflare Audit Log

#### Problem

The Cloudflare interface does not make it easy to get a quick overview of our current rules applied to each zone and compare them for consistency. There is also variation due to configuration required for production that may not be necessary for staging.

#### Proposal

Cloudflare firewall rules described by issues in gitlab-com/gl-infra/cloudflare-firewall> should be labelled with the zones they are applicable to and checked by Cloudflare Audit log.

The current proposed labels from https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2783/diffs#5a3efee5111997f9405246307d030ade2f15d696_148_152

```markdown
  - `zone:gitlab-com`
  - `zone:staging-gitlab-com`
  - `zone:gitlab-net`
```

#### Potential Tests

- [x] Add the necessary labels to existing open tickets
- [ ] Update Cloudflare Audit to check for the rules in each zone
- [ ] Update Cloudflare Audit to check if the matching rules in each zone they are in are identical
- [ ] For the rules that exist in multiple zones, verify that they are in the same order.

cc @T4cC0re @cmcfarland @dawsmith for feedback and discussion.

Story points: 3.0
Create new gitaly storage shard node to replace `nfs-file49`

Gitaly storage shard `nfs-file49` (`file-49-stor-gprd.c.gitlab-production.internal`) is at `85.07%` usage as of `2020-09-25`.

Our usage targets specify that we try to maintain usage between 65-79%.  New project creation would quickly cause more usage than that on `nfs-file49`.

There are currently 6 gitaly shard nodes configured to accept new projects (`nfs-file49-55`).  Maintaining at least 7 or 8 shards with sufficient capacity for new user repository creation is about the target level of availability we prefer and is important because it helps us avoid a scenario in which shards fill up too quickly.

Note: I have already removed `nfs-file49` from the configuration to receive new repositories, because it has become well past the 80% threshold by which it should have been removed already.

To remove a single node from the new projects storage rotation cluster, and also prevent the acceleration of capacity consumption, a new gitaly shard node should be created and added to the list of shards configured in the GitLab Application to store new project repositories.

[Create a production change issue for this, using the `storage_shard_creation` template: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/new?issuable_template=storage_shard_creation)

Story points: 3.0
Set up backups, delayed replication, and snapshots in staging

We should define for the registry database cluster in staging:

**backup pipelines**

 - We execute a daily backup and copy the wal files daily. This is triggered by a shell cron-job that we have in chef. We need to adapt to be generic on the database name.
 - We have a restore pipeline to verify if the backup can restore properly.
 - Enable our deadmansnitch alerts for the restore of this new database backup restore.
 - review our alerts on the backup generation and alerts.

**delayed replica**

 - As we have in GitLab.com, we need a delayed replica, where we apply the WAL files with a delay of 8 hours. The main intention of this database is to restore data in case of accidents, e.g.: if we had an accidental `DELETE` statement, or some wrong `WHERE` clause in a `DELETE` statement in production,  we would have an environment to recover the deleted data quickly.
 - This host is not part of the Patroni cluster.

**Disk snapshots**

- We want to have a dedicated replica that will not receive traffic and can not be promoted to be the leader of the Patroni cluster for the container registry database cluster.
- As we execute in GitLab.com this snapshot has to have the following steps to guarantee the synchronizd:
  * Execute a pg_Start_backup before the snapshot is initiated.
  * execute a GCP disk snapshot
  * Execute a pg_stop_backup after the snapshot is finished.

All the retention policies should be as GitLab.com.


Acceptance criteria: 
- [ ] Fulfill all the requirements from the issue

Story points: 13.0
Create the (new) monitoring for the Container Registry DB in production

We should review our monitoring set up in a generic method that we could deploy and monitor several clusters, without impact each cluster.

The metrics and alerts could be customized per cluster since the business requirements can be different. 

Also, we should review our tools, ( scripts, ansible-playbooks...) that we could test and refactor if needed, to be used in different clusters.

Story points: 8.0
Connect Smartling to Brand and Digital Design team Repository

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 
 - Point of contact for this request: [+ @sdaily +]
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**Documentation**

Verify the following system prerequisites are installed: 

- Repo user created for Repo Connector
- The application should be hosted on a server that is continuously available and/or publicly addressable.
- Java version 8 or higher.
- Disk space requirements: It should have enough space to clone your Git repositories, as well as 50MB for the installation of the Repository Connector.
- Links to relevant repos (if public)
- Example resource files (e.g. GetText .PO file, JSON file, etc)
- Help Center articles on the Repo Connector: https://help.smartling.com/hc/en-us/sections/360001685234-Repository-Connector

**SRE Support Needed**

*dsmith - from call notes, we just need to:

1. Create a Smartling GCP Project under the Marketing Folder
1. Setup a VM in the new project -  n1-standard-2 should do, ubuntu 18.04 or 20.04 - setup auto-patcthing with 100GB disk to start.
1. Do basic hardening
1. Set up a way to maintain ssh keys for the marketing team to access the config files
1. Install the connector per the docs above.

Open question, should we set up a simple ansible repo to do the hardening/key management (or okta asa)/connector install?

## Related MRs

- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/114
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/117
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/119
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/120
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/123
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/124
- https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/merge_requests/128

/cc @mpreuss22 @bmatturro @brandon_lyon @laurenbarker

Story points: 8.0
Investigate to turn off Cloudflare E-Mail Obfuscation for performance reasons

Current on gitlab.com an JS is loaded automatically through Cloudflare which is responsible for a feature called E-Mail Obfuscation, which is there to protect e-mail adresses from being scraped. It seems that this was not deliberately turned on but rather is a default. Would be great to investigate and/or turn the feature off to gain some performance through that. 

JS that is injected on each page through Cloudflare automatically: https://gitlab.com/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js 

Example page where it is loaded: https://gitlab.com/gitlab-org/gitlab

Cloudflare info: https://support.cloudflare.com/hc/en-us/articles/200170016-What-is-Email-Address-Obfuscation-

Story points: 2.0
New Readiness Review for Container Registry + new PG DB cluster

The new Infra Readiness Review process and template are being created and validated [here](https://gitlab.com/gitlab-com/gl-infra/readiness/-/merge_requests/43). We'll need to follow these two processes/templates:
- [Operational readiness](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/13c0b59a3e61af6993084be786a06c74199bab51/.gitlab/issue_templates/service_readiness.md)
- [Production Readiness](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/13c0b59a3e61af6993084be786a06c74199bab51/.gitlab/issue_templates/production_readiness.md)


Critical points to fully cover in this readiness review are:
- [ ] Complete our DB Runbooks - taking into account our new DB clusters (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11321)
- [ ] Update Monitoring (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11326)
- [ ] A robust Backup setting

(This issue is still work in progress)

Story points: 8.0
Cloudflare WAF Rules for `/api/graphql` and `/-/wikis`

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 
 - Point of contact for this request: [+ @asaba +]
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**SRE Support Needed**
[+ Support Request Details +]

The following paths should be excluded from the WAF rules:

* POSTs to `/api/graphql`, exact match
* POST to paths containing `/-/wikis/`
* POST to paths ending with `/-/wikis`

Based on the most recent traffic analysis, these are the last exceptions I think we should add before enabling the WAF in block mode. Details here: https://gitlab.com/gitlab-com/gl-security/appsec/appsec-team/-/issues/50#note_412543041

<!--
please do not edit the below
-->

Story points: 3.0
New DB for Container Registry: Create the DB monitoring in Staging

**Metrics, dashboards and alerts**
Our goal is to collect all the metrics to have the following dashboards for the container registry:

- Patroni overview: https://dashboards.gitlab.net/d/patroni-main/patroni-overview?orgId=1
- Postgresql overview: https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1
- Pgbouncer overview: https://dashboards.gitlab.net/d/PwlB97Jmk/pgbouncer-overview?orgId=1
- Host stats: https://dashboards.gitlab.net/d/bd2Kl9Imk/host-stats-old-prometheus?orgId=1&var-environment=gprd&var-node=patroni-01-db-gprd.c.gitlab-production.internal&var-promethus=prometheus-01-inf-gprd 

Initially, the alerts should be equal to GitLab.com. If we need we will adapt them.

**Logs**

We would like to ensure that we have all the logs from the database ecosystem(PostgreSQL, Patroni, PgBouncer and Consul) available in:

* ELK - Kibana

* Sentry

Story points: 13.0
Provision a new database cluster in production for the container register service

We need to provision a PostgreSQL cluster in production, after our staging tests are completed. This issue will address the change design and preparation (Design/validation sessions, Change testing, create documentation), with the below requirements.

Requirements:

* Install PostgreSQL version 12.
* The hardware specs have to be evaluated with the traffic that we are expected.
* Create a cluster with 1 primary and 2 nodes read-only.
* Configure a new Patroni cluster.
* Create a new cluster in consul (that Patroni will use)
* Provision 2 pgbouncers one to receive the traffic Read-write and other one to receive traffic read-only.

Story points: 8.0
Memory troubleshooting guide

(Coming from this incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2581)

Diagnosing memory issues is notoriously hard. There are several methods that can be used, and they are largely dependent on the type of process being profiled.

It would be good to have a guide that goes into how to diagnose high memory usage.

It may make sense to have separate guides per process type.

* Go: We have pprof and potentially even continuous profiling.
* Ruby: Stackprof, heap dump via objspace. GC stats. jemalloc stats.
* Generic: pmap, smaps, core dump, heap dump via gdb.
* Novel: [heaptrack](https://github.com/KDE/heaptrack), [tcmalloc](https://github.com/google/tcmalloc), [poireau](https://github.com/backtrace-labs/poireau).

Some of these need some more work on the tooling side on our end. But at the very least pprof and continuous profiling would be good to document.

refs https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/84

Story points: 1.0
Provision new Patroni cluster in staging for the Container Registry service

We need to provide a PostgreSQL database cluster in staging.

### Status

* **MRs**
  * **DBTB** 
    * Chef: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/33
    * TF: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/2581
  * **Staging** COMPLETED
    * Chef: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/75
    * TF: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/2615
    * CR: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4796

Requirements:
- [x] Install PostgreSQL version 12.
- [x] Hardware config `n1-standard-8`, or we could migrate to `n1-standard-16`
- [x] Create a cluster with 1 primary and 2 nodes read-only. One of them dedicated to snapshots.
- [x] Configure a new Patroni cluster.
- [x] Create a new cluster in consul ( using the same cluster we use for GitLab.com ).
- [x] Provision 2 pgbouncer pools, one to receive the traffic Read-write and the second to receive traffic read-only.
- [ ] We want to offer by default redundancy on pgbouncer. We will have at the moment 2 instances to balance the load on the pgbouncer level since pgbouncer is single-threaded, and in case of a failure we would redirect the load on the second instance. 
- [ ] If we need to expand we would use 2 CPU-cores from each node from the pgbouncer layer that are in front of the primary database host, and would not impact performance on the database layer. Imagine from a hardware perspective form the primary with 32 cores having 4 cores dedicated for pooling, is over 10% that I consider high. And we would be able to scale up this layer.
- [ ] if we want to use in the future some functionality as pgbouncer pause, it is ideal to have the pgbouncer nodes on a different level, principally considering a hardware problem where the pgbouncer could be impacted with the database node. 
- [x] We need to offer 2 pgbouncer pools, from read-write and read-only, to enable good practices from our developers to redirect the read-only traffic to the secondaries.
- [x] Let's enable data checksums in the database from the initial setup.

Story points: 21.0
Add restore_command to recovery.conf in patroni cluster

Adding `wal-g wal-fetch` as a `restore_command` to the recovery.conf in our patroni cluster would help with catching up from a big replication delay by loading wal files from GCS until we are close enough to switch over to streaming replication (which will be done transparently by postgres).

This would reduce the load on the primary and also enable recovering when WAL files already are missing on the primary.

We already have wal-g binaries installed on all patroni nodes in gstg and gprd. But while gstg will work out of the box, because we use wal-g for wal-push there already we need to point the wal-g configuration in gprd to the wal-e GCS bucket, as we did not enable wal-g wal-push there yet.

This is a corrective action for https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2570.

Story points: 5.0
disable-chef-client isn't preserved over reboots

In https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2570#note_401672637 we saw that disabling chef-client with `disable-chef-client` wasn't preserved over a reboot - which unexpectedly brought the DB back into rotation with a high replication delay.

We need to figure out why it isn't preserved over reboots (the node was in an unhealthy state, so maybe it just failed to write changes to disk in this one case and works in general - need to test) and fix it.

Story points: 3.0