giseldo/neodataset huggingface dataset:
Make CloudFlare workers terraformable

In order to deploy a prototype for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8209#note_238224109, we need to have a way to deploy cloudflare worker scrips and routes vie terraform.

This issue's goal is to have a way to deploy worker scripts (https://www.terraform.io/docs/providers/cloudflare/r/worker_script.html) and worker routes (https://www.terraform.io/docs/providers/cloudflare/r/worker_route.html) to staging.gitlab.com.

Story points: 3.0
Use `web_exporter/metrics` for staging and production.

The `/-/metrics` is a very bad citizen now when we run Puma. It does effectively block the worker process for the time required to render all metrics, thus increase the latency and processing time across the fleet substantially: https://dashboards.gitlab.net/d/thYzurImk/rails-controllers?orgId=1&var-action=MetricsController%23index&var-database=influxdb-01-inf-gprd. We do run around 1k of `/-/metrics` requests per-minute, the mean time is 400ms, p95 1s. Anything that escapes to native, is not interruptible by default due to Ruby GVL and effectively blocks the requests processing for the given period of time.

Looking at actual implementation, it is actually very inefficient: https://gitlab.com/gitlab-org/prometheus-client-mmap/blob/master/lib%2Fprometheus%2Fclient%2Fformats%2Ftext.rb#L34. We get a list of all files, and escape to native to present them.

We do have `/metrics` on implemented already on separate endpoint, it is not yet fully tested, but I would assume that we might prefer to switch to that: https://gitlab.com/gitlab-org/gitlab/issues/30037.

We should validate and switch it on staging/production to use `/metrics` endpoint.

Story points: 3.0
Environment name for new ops environment

When provisioning the new Chef server infrastructure, we received the following error on bootstrap: `chef-01-sv-ops-us-central startup-script: INFO startup-script: Cannot load environment ops-us-central#033[0m` because we've overloaded that variable/term.

At first glance, I was inclined to set this value to `ops` in `environments/ops-us-central/variables.tf`, thinking that our intention is to lift-and-shift the current ops environment to the new region, destroy the old infrastructure, then update the terraform config directory name and state file to `ops` and let it be the new primary. However - currently there are a few areas where `ops-us-central` is hard-coded that might still be problematic, and prevent this approach; most notably [the name for the GCP network](https://gitlab.com/gitlab-com/gitlab-com-infrastructure/blob/d2f51e5d709943cad25a305c73a6eb4f69581b90/environments/ops-us-central/main.tf#L52), and subsequent references under downstream dependent resources.

In order to facilitate the migration, I anticipate that we will be limited in our options for names we can use while still effectively running both environments in parallel, but `ops-us-central` is likely not the best choice, in terms of length, and region-specificity to start; I'm sure that we'll encounter other issues as we continue, as well. `ops-too` was used for the first aborted migration, but that received pushback due to being somewhat flippant.

Since we're restarting the efforts to move ops away from the `us-east1` region, I suspect now is the time to figure this out, while we can reasonably easily destroy/recreate infrastructure without impacting running services. If we _do_ have to use an entirely new name, that means duplication of our chef configs, at least; what else would be impacted?

TL;DR - what shall we call the new ops environment / how do we address the naming issues? 

/cc @ahanselka @cmcfarland @cindy

Story points: 1.0
Thanos compact disk filling up

The `/opt/prometheus` disk on `thanos-compact-01-inf-gprd.c.gitlab-production.internal` is almost full.

This is similar to this issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6776

The alert was cleared by following the runbook here: https://gitlab.com/gitlab-com/runbooks/blob/master/troubleshooting/thanos-compact.md

The following commands were executed:

```
df -h
sudo systemctl stop thanos-compact
sudo systemctl start thanos-compact
df -h
```

/cc @bjk-gitlab

Story points: 1.0
Validate scraping of Praefect on gstg

Staging has Praefect enabled, though I couldn't find the metrics in Prometheus. One metric I looked for specifically is `gitaly_praefect_replication_latency`. That one is always created, regardless if it has input. Given it was missing I think we're not scraping the target, or we're not exposing metrics on the target.

/cc @alejandro

Story points: 2.0
Intermittent HTTP failures to Ingress controllers

We have been seeing intermittent HTTP failures when trying to connect to ingress controllers.  

The first instance was the `design.gitlab.com` site in this issue: https://gitlab.com/gitlab-org/gitlab-services/design.gitlab.com/issues/432

That was resolved by upgrading the K8s cluster and provisioning a new ingress, then re-deploying the app to it.  We tried provisioning a new ingress alone, and it didn't resolve it.

The second instance is some suspicious failures on the `version.gitlab.com` app.  We got a Pingdom check failure, and `httping` is returning some intermittent timeouts.

These may not be related, but this issue can be a place to capture information and explore the possibility.

```
$  httping -c 15 -t 2 http://version.gitlab.com
PING version.gitlab.com:80 (/):
connected to 104.196.17.203:80 (208 bytes), seq=0 time=294.46 ms
connected to 104.196.17.203:80 (208 bytes), seq=1 time=293.93 ms
connected to 104.196.17.203:80 (208 bytes), seq=2 time=292.10 ms
connected to 104.196.17.203:80 (208 bytes), seq=3 time=296.31 ms
connected to 104.196.17.203:80 (208 bytes), seq=4 time=293.10 ms
connected to 104.196.17.203:80 (208 bytes), seq=5 time=294.27 ms
timeout while receiving reply-headers from host
timeout while receiving reply-headers from host
connected to 104.196.17.203:80 (208 bytes), seq=8 time=293.68 ms
connected to 104.196.17.203:80 (208 bytes), seq=9 time=294.75 ms
connected to 104.196.17.203:80 (208 bytes), seq=10 time=293.74 ms
connected to 104.196.17.203:80 (208 bytes), seq=11 time=295.00 ms
connected to 104.196.17.203:80 (208 bytes), seq=12 time=361.12 ms
connected to 104.196.17.203:80 (208 bytes), seq=13 time=301.54 ms
--- http://version.gitlab.com/ ping statistics ---
15 connects, 12 ok, 13.33% failed, time 22342ms
round-trip min/avg/max = 292.1/300.3/361.1 ms
```

Story points: 3.0
Documentation for new Chef Infra Server

Broken out from #8037; the infrastructure has been provisioned, and in conjunction with / once #8028 is completed, we need to document the changes to the infrastructure, and conduct a [readiness review](https://gitlab.com/gitlab-com/gl-infra/readiness/-/issues/11) before proceeding with a production change for the migration. This issue should include discussion centered around documentation and lead into the production readiness review.

Story points: 3.0
Implement/update monitoring for Chef server in GCP

Broken out from #8037; the infrastructure has been provisioned, and in conjunction with / once #8028 is completed, we need to implement [monitoring](https://docs.chef.io/server_monitor.html) for the Chef Infra Server application.

Definition of done:
* up check on port 443
* hook up the node exporter
* make a basic jsonnet def in the dashboards

Story points: 2.0
Update cookbook-gitlab-runner to use new Docker packages

[cookbook-gitlab-runner](https://gitlab.com/gitlab-org/cookbook-gitlab-runner) is still using the obsolete `docker-engine` package repo. This should be updated to use the new [docker-ce package](https://docs.docker.com/install/linux/docker-ce/ubuntu/) repo.

CC: @jarv @tmaczukin

Story points: 3.0
Facilitate 2nd Failover Test of Staging Using Geo With Learnings from 1st Test Applied.

As part of the deliverables of the [GitLab.com Disaster Recovery Working Group](https://about.gitlab.com/company/team/structure/working-groups/disaster-recovery/) we would like to conduct a 2nd single-node test of a Geo-enabled failover for staging.gitlab.com.

This test would be a planned failover of staging at a scheduled time, which:
1. Fails over staging.gitlab.com to a Geo-based secondary
1. Tests that staging.gitlab.com works on this secondary
1. Fails back over to the original staging.gitlab.com infrastructure.

What needs to happen to facilitate this:
- [ ] Apply changes and learnings from the first single-node test.
- [ ] Build a change management issue to track the tactical pieces to this failover.
- [ ] Coordinate with various consumers of staging to help define a testing schedule and what we will need to have in place in the event we are without staging for an extended period of time

Story points: 8.0
Modify Fastly headers to accommodate same-origin content in frames

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details** 

 - Point of contact for this request: @djensen
 - If a call is needed, what is the proposed date and time of the call:  n/a
 - Additional call details (format, type of call): n/a

**SRE Support Needed**

Please modify Fastly headers for about.gitlab.com to allow for `<object>` tags to be used with same-origin content (to allow [clickable SVGs](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10052#proposal)):

1. Remove the `X-Frame-Options` header.
1. Add the CSP `frame-ancestors` directive, and whitelist `about.gitlab.com`.

This proposal was already discussed here: https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10052#note_458792328

<!--
please do not edit the below
-->

Story points: 1.0
PoC: rails-console read-only sandbox

Based on https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11730#note_455197827

> I have a rough idea of how we can sandbox the read-only console pretty well.
> 
> We can force SSH to not start a shell, but rather a transient systemd-unit that is forwarded via SSH. That we can lock down as tight as we need to (filesystem ro, partially ro, even network access can be sandboxed to specific hosts). By forcing every user to access the rails-console via that, we can make sure, that whatever we configure is confined in the sandbox and the appropriate config files would be to.
> 
> We should probably have one designated write-only target though if someone needs to export something. But if the user is unprivileged, they would need the interaction of another person to get the exports. That way we can also eliminate possible scenarios where someone might have gone rouge (which hopefully never happens!)  
That way we can also prevent users from shelling out of rails into the OS. We can either restrict it, so they can't shell out period, or they can fiddle around in their sandbox.
> 
> The read-write console access might be unsandboxed I think.

Scope:
- Enforce full read-only access to *all* resources outside the sandboxed instance.
- Check feasibility of running sandboxed rails-consoles.
   - What infra does rails need, that also requires read-only instances?
      - Can we force rails to treat those (except Postgres - where we most likely *will* rely on a replica) as read-only (/force not triggering writes)?
   - Does rails care if the state it writes to disk is not carried forward to the next startup?
- Consider how files exported by rails-console could be extracted from the machine and ensure those ways are known and regulated.
- All changes and how they are derived *must* be documented in this issue.
- Ensure output of the sandbox is captured in the systemd journal so it can be easily logged.
- Codeify changes in chef if possible.

Not scope:
- User access and group mappings. This PoC assumes allowed users are in the group `rails-console-read-only` and can authenticate to the ssh daemon.

Story points: 20.0
Disable wal-e in gprd

After running wal-g successfully in parallel to wal-e since a while in gprd, and gstg fully being [switched over to wal-g](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3077) already,  we can disable wal-e in gprd after making sure we [switched over all wal-e backup consumers in gprd](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11983).

* [x] [switch all wal-e backup consumers to wal-g](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11983)
* [x] add alert silence for wal-e
* [x] switch archive_command on primary to use wal-g, disable `wal-g wal-push` on secondary
  * [ ] merge MR for disabling wal-e
  * [ ] check archive_command on primary
  * [ ] switch archive_mode from `always` to `on` on patroni-08 (needs postgres restart)
  * [ ] disable archive_lock cronjobs on all nodes
* [x] disable wal-e backup-push cronjobs on all nodes
* [ ] cleanup chef roles
* [ ] update runbooks

For execution we can make a copy of the CR issue for gstg, which worked well: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3077

## Acceptance criteria:

* [x] all backup and wal file consumers are configured to read from the wal-g archive (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11983)
* [x] wal-e is disabled in gprd

Story points: 3.0
switch all gprd db replicas to restore from wal-g archive

We now have working and tested daily db backups done by wal-g in gprd and gstg has wal-e completely disabled already. To be able to disable wal-e in gprd, we need to find all current consumers of the wal-e archive and switch them over to use the wal-g archive.

For https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11852.

* [x] archive replica
* [x] delayed replica
* [x] database-lab/joe
* [x] any active gitlab-restore instances
* [x] make wal-g archive the default path for new restore instances
* [x] anything else?

## Acceptance criteria

* [x] All backup consumers use wal-g archive by default

Story points: 5.0
disable wal-e in gstg

After running wal-g successfully in parallel to wal-e since a while in gstg and gprd, we can disable wal-e in gstg after making sure we [disabled all wal-e backup consumers in gstg](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11883).

* [x] gstg: [disable wal-e backup consumers](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11883)
* [x] gstg: add alert silence for wal-e
* [x] gstg: switch archive_command on primary to use wal-g, disable `wal-g wal-push` on secondary
  * [x] merge MR
  * [x] check archive_command on primary
  * [x] switch archive_mode from `always` to `on` on patroni-05 (needs postgres restart)
  * [x] disable archive_lock cronjobs on all nodes
* [x] disable wal-e backup-push cronjobs on all nodes

Story points: 3.0
Investigate Failed builds for `gitlab-com/www-gitlab-com`

**NOTE: Much of the initial discussion and links to failed jobs are contained in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3104.  TODO: Any relevant/necessary info should be moved to this issue, then that one can be closed as a duplicate.**

@cwoolley-gitlab Reported to us a couple of failed jobs when cloning or fetching objects for the `www-gitlab-com` repository.

- https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/863088257
- https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/862981337

I'm pretty sure that I found the error for the first job.

```
Job failed (system failure): Error response from daemon: error setting label on mount source '/var/lib/docker/volumes/runner-d5ae8d25-project-278964-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70/_data': bad message (docker.go:817:4s)
```

Link to document:

https://log.gprd.gitlab.net/app/kibana#/discover/doc/AWgzayS3ENm-ja4G1a8d/pubsub-runner-inf-gprd-000339?id=dZoV53UBQEXcWSi0HOqw

Its duration is `1,377,525,329,393` which I presume is nanoseconds, and this translates to `22.958755489883334` minutes, which roughly correlates with the error message from the job: `ERROR: Job failed: execution took longer than 20m0s seconds`.

There may be some relationship to a couple on-going issues being tracked separately:

* https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/1358#note_451945972
* https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3068

This issue is blocked by [another issue to gather the correct metrics](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12213)

Story points: 2.0
Identify deliverables for beta launch of macOS Shared Runners on .com

The macOS Build Cloud (runners on .com) [closed beta](https://gitlab.com/groups/gitlab-org/-/epics/3922) is currently in progress. The goal over the next few milestones is to wrap up the work on the autoscaler to transition the solution to [open beta](https://gitlab.com/groups/gitlab-org/-/epics/3926). The target for the open beta launch is milestone 13.10, March 2021.

In addition to the [autoscaling](https://gitlab.com/groups/gitlab-org/-/epics/3936) development, as we learned with the Windows Shared Runners rollout, there is significant work required by the infrastructure team that is a pre-requisite to the open beta launch.  This issue is to start the discussion and planning needed to get the infrastructure components in. place.

Story points: 3.0
switch all gstg db replicas to restore from wal-g archive

We now have working and tested daily db backups done by wal-g in gstg. To be able to disable wal-e in gstg, we need to find all current consumers of the wal-e archive and switch them over to use the wal-g archive.

For https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11852.

* [x] archive replica
* [x] delayed replica
* [x] geo secondary db
* [x] database-lab/joe (?)
* [x] any active gitlab-restore instances?

Story points: 5.0
Update cirepom to leverage new gitlab.com repository_storage_moves API

**Update 2020-11-13**: This issue is being closed, since it does not appear that this will be a viable option, given the ostensibly short term of the impending application-based dog-fooded solution.

The more I look into modifying the `cirepom` project, the more I become concerned that the status quo is not aligned with the state of the art from the GitLab project repository storage move api.

I think I might need to develop a better understanding of how the code base operates, and how it integrates with gitlab.

Currently, it appears that `cirepom` initiates a repository migration in this manner:

```ruby
def set_project_repository_storage(id,target)
  exec :put, "projects/#{grok id}", "repository_storage=#{target}"
end
```

Source: https://gitlab.com/gitlab-com/gl-infra/cirepom/-/blob/master/lib/cirepom/store/gitlab.rb#L31

And then subsequently monitors and tracks state, quarantining projects whose migrations exhibited errors.

This is problematic for two reasons:

1. The gitlab application now tracks migration state itself and exposes an API to monitor the state of a given migration.
2. The gitlab application project repository storage move API has significantly reduced error rates for migrations and increased stability to the point that the quarantine features here may not be necessary, since it is my understanding that they are designed to prevent repeated attempts to migrate repositories whose migrations had already failed.

New `gitlab.com` API is at `projects/<project_id>/repository_storage_moves`.

It doesn't even appear that the defacto GitLab client ruby gem has support for this API, either.  As of 2020-11-10, there are no results for: https://github.com/NARKOZ/gitlab/search?q=repository_storage_moves  But maybe this is a naive query.

It is unclear to me whether the Firestore datastore features surrounding migration state management and quarantining are of much use after such an update.

Story points: 4.0
Renew fuzzit.dev

Renew fuzzit.dev domain for 4 years
We completed the acquisition of FuzzIt this past year and transferred the domain fuzzit.dev to GitLab.

The website will be deprecated but we want to add 4 years to the registration to prevent anyone from registering the domain and benefitting from the brand equity.

Please add 4 years to the fuzzit.dev domain.

Story points: 1.0
Include statements in postgresql elastic logs for better debugging

Currently, for analysing slow queries, we need to log into a DB host and search through the local logs, because we redacted query statements when sending logs to elastic, for security reasons.

Instead we should consider to include the statement in the logs and just redact values between `''`. This should allows us to aggregate by class of query and make it easier to debug DB performance issues because of slow queries.

~"corrective action" for https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2885.

Story points: 3.0
Alert if a small set of queries is dominating postgresql

Normally, most of the DB workload is spread out over many different queries. If the postgres workload is dominated by only a few slow queries (because of missing indices or statistics leading to bad query plans), this can have a severe impact on overall DB performance - up to a full downtime of GitLab.com as seen in [this incident](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2885) - but is sometimes is hard to detect, as we will get many symptomatic alerts but no alerts directly pointing at slow queries.

We should alert if only a small set of queries is dominating the total query time, as suggested [here](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2885#note_436200513).

~"corrective action" for https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2885.

Story points: 3.0
Create terraform repo for transient-import project

We get requests often to make VMs for customer project imports (ex: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11695). As of now, these are manually provisioned by someone in infra and handed off to the support or professional services engineer performing the import. I think we could be much more efficient if we made a terraform repo and allowed engineers to mostly self-service the creation of servers.

I propose that we re-use the [GCP module in dev-resources](https://gitlab.com/gitlab-com/dev-resources/-/tree/master/modules/gcp) to make this possible. This module was written for support to create VMs in a similar way for interviews or other testing purposes. The main thing we would need to modify is to remove the chef provisioning part and add a firewall rule to the module to allow the support engineer to access via SSH. 

Since much of the terraform is done, this shouldn't take too long to do and would make both infra and support more efficient in these requests.

Story points: 5.0
Install the `pg_checksums` debian package on patroni nodes

To support postgres [checksums enablement](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11518#note_436803777), install the `pg_checksums` debian package on patroni nodes from the gitlab Aptly package host.

Package is: `gitlab-pg-checksums_1.0_amd64.deb`

It should already be uploaded to aptly, but I'll attach it here, just in case.

[gitlab-pg-checksums_1.0_amd64.deb](/uploads/30539087472549bdbbe9143249a97c53/gitlab-pg-checksums_1.0_amd64.deb)

The goal of this issue is implement how the package that is already in Aptly will be rolled out using chef in the database nodes.

Story points: 1.0