giseldo/neodataset huggingface dataset:
Metrics and dashboards improvements

## Context

This is part of the work to upgrade and migrate the GitLab.com container registry to a new version backed by a metadata database and online garbage collection (https://gitlab.com/groups/gitlab-org/-/epics/5523). This will be achieved following the gradual migration plan detailed in https://gitlab.com/gitlab-org/container-registry/-/issues/374.

## Task

Go through the list of Prometheus metrics and Grafana dashboards for the registry and ensure that everything is working as expected and displayed accurately. This is a good opportunity not only for fixes but also for generic improvements.

Story points: 3.0
Clean up Consul node snapshots after OS upgrade

After upgrading the console nodes https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5481 and https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5606, there will be some snapshots which we want to keep for at least several days, but probably not more than a few weeks.  We want to be able to close out the upgrade CR when it is finished, and not keep it open pending deleting these snapshots.

This issue is a reminder to clean up the snapshots after a reasonable amount of time has passed.

Story points: 2.0
Increase the `shared_buffers` of `postgres-dr-delayed-01-db-gprd.c.gitlab-production.internal` to `20GB`

Increase the `shared_buffers` of `postgres-dr-delayed-01-db-gprd.c.gitlab-production.internal` to `20GB`.

See: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5428#note_663616379

Story points: 1.0
Upgrade GitLab services to terraform v1.0.X


The latest major version of terraform (v1.0) has hit GA and we should start planning to upgrade to avoid unnecessary tech debt.

We will have to push the changes to the following Auto-Devops environments:

- [x] services-staging
- [x] services-prod
- [x] license-stg
- [x] license-prd
- [x] gs-staging
- [x] gs-production


Related Issues:
- We can use #12287 (closed) as a procedure reference.
- [gitlab-com-infrastructure Terraform v0.14 to v1.0.4 Upgrade](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13908)

Story points: 2.0
Remove un-needed keys from the staging omnibus GKMS vault

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5050

Per this incident issue, we should take a short term corrective action to fix the immediate problem of the Omnibus GKMS vault file being too large to encrypt. There are other on-going efforts to compress these JSON files before encryption.

Per @alejandro, we should be able to remove some keys for praefect TLS to help save enough space to get things working in the short term. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5050#note_617653560

Specifically, removing these keys:
- `omnibus-gitlab`:
   - `ssl`:
      - `praefect_certificate`
      - `praefect_private_key`
      - `trusted_certs`:
         - `praefect.crt`

Story points: 2.0
Validate an existing node can switch to the v10 boot script

Because of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13477 we can update the boot script and just require a reboot for it to stick.

The v10 of this script introduced in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13566 forces a node to use a `-generic` kernel to receive Ubuntu Advantage Kernel Livepatches.

We should validate that we can switch a node to v10 without a rebuild, by just rebooting.

AC:

- [x] In gstg create a new instance of `generic-stor` for each boot script version prior to v10 and each Ubuntu 16.04, 18.04, and 20.04
   - start version is debatable, check in TF what is the lowest version in use.
- [x] Make sure everything runs as usual (chef-client works successfully)
- [x] bulk update the script version to v10 in TF
- [x] reboot the machines
- [x] Check the machines are now running a `-generic`-kernel
- [x] Make sure everything runs as usual (chef-client works successfully)

Checklist / notes whether it worked: (Replace `OK` with notes if required)

<table>
<thead>
<tr>
<th>Script version/OS</th>
<th>xenial / 16.04</th>
<th>bionic / 18.04</th>
<th>focal / 20.04</th>
</tr>
</thead>
<body>
<tr>
<td>
6
</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
</tr>
<tr><tr>
<td>
7
</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
</tr>
<tr><tr>
<td>
8
</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
</tr>
<tr><tr>
<td>
9
</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
</tr>
<tr><tr>
<td>
 10 itself (tested in dev)
</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
<td>

- [x] OK

</td>
</tr>
<tr>
</tbody>
</table>

(Any failures in regards to the epbf-exporter can be ignored, as long as chef-client still finishes. This is a known issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13575)

Story points: 8.0
Ubuntu Advantage aware bootstrap script

This issue is to split our scope-creep from https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10039 regarding https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/bootstrap/-/merge_requests/14

As per https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10039#note_577158689:

Rework kernel management
  * Install the latest kernel at node bootstrap time, no more hardcoding of versions. (make sure apt holding works though, just in case we are forced to rollback a kernel)
  * Use livepatch to keep security updates
  * Use livepatch status to see whether all security updates could be applied or if there is a reboot required.

The MR linked above adds it. This issue is to document the rollout of v10 of the bootstrap script.

Story points: 8.0
Teleport approvals not working from remote command lines

The `tctl` command is not working from remote sessions.  It works fine from the server, but from an approver's laptop, it's not working.

The error looks like: 

```
[CLIENT]       Cannot connect to the auth server: failed direct dial to auth server: Get "https://teleport.cluster.local/v2/configuration/name": x509: certificate is valid for teleport.gprd.gitlab.net, not 73746167696e672d74656c65706f72742d636c7573746572.teleport.cluster.local
    Get "https://teleport.cluster.local/v2/configuration/name": x509: certificate is valid for teleport.gprd.gitlab.net, not 73746167696e672d74656c65706f72742d636c7573746572.teleport.cluster.local, failed dial to auth server through reverse tunnel: Get "https://teleport.cluster.local/v2/configuration/name": dial tcp 34.74.121.190:3024: connect: operation timed out
    Get "https://teleport.cluster.local/v2/configuration/name": dial tcp 34.74.121.190:3024: connect: operation timed out.
Is the auth server running on "teleport.gprd.gitlab.net:3080"?
```

Story points: 2.0
Container Registry: Define custom SLI for blob upload routes

## Context
As part of https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/476, we're breaking the single Container Registry SLI into multiple SLIs per API route.

## Proposal

Separate the blob upload routes SLI. 3 SLIs is what we should go for this route:

* `server_route_blob_upload_uuid_writes`: `PATCH`, `PUT`
* `server_route_blob_upload_uuid_deletes`: `DELETE`
* `server_route_blob_upload_uuid_reads`: `GET` (with a comment that `GET` is unused?)


See https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/476#note_578464890 for additional details.

Story points: 2.0
Investigate/setup restart policy for teleport

Discovered on staging today (2021-05-13) that teleport on staging had been dead for a while:

```
dsmith@teleport-01-inf-gstg.c.gitlab-staging-1.internal:~$ systemctl status teleport
● teleport.service - Gravitational Teleport - Access Control Server
   Loaded: loaded (/etc/systemd/system/teleport.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Fri 2021-04-30 21:15:51 UTC; 1 weeks 5 days ago
     Docs: https://goteleport.com/teleport/docs/
 Main PID: 19722 (code=exited, status=0/SUCCESS)
```

Oddly the code is exited / success so I wonder what happened.

Looks like the systemd unit does have `Restart = on-failure`, but that didn't work in this case with success?

Story points: 2.0
Disk space usage on nfs-file-{48,49,50,51} is between `86.41%` and `89.45%`

We might need to do some repository shuffling away from `nfs-file-{48,49,50,51}` which are between `86.41%` and `89.45%`: https://dashboards.gitlab.net/d/W_Pbu9Smk/storage-stats?orgId=1&refresh=30m&var-env=gprd&var-environment=gprd&var-node=All&from=1620742126000&to=1620828526000&viewPanel=160

Unless there is some trigger which automatically re-homes git repositories after a specific threshold is crossed.

This is a scenario which this epic (https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/357) is intended to address, and is implementation complete (https://gitlab.com/gitlab-com/gl-infra/balancer), fully tested in `staging`, but not yet tested in production.

Obviously, we can do this manually very easily now.

I think it is worth discussing if now would be a good time to proceed with production testing of the automation.

Story points: 1.0
Setup ansible user across the fleet

In order to support testing Ansible against nodes in our GCP projects, we will need to have a target user setup for connections from the control node (CI runner).

To establish access to existing hosts, we will create a data bag for the user in Chef, and add a pre-generated SSH key (that will be saved in 1Password).

For long-term access that will remain once we move further with the migration and no longer have Chef, or no longer perform user provisioning with Chef, we will need to update the bootstrap module to make sure the user exists when a compute instance is initially provisioned by terraform.

Story points: 3.0
Developer Evangelism Group Project

# Group Project Request

- Project / Group Name (<17 characters and start with `group-`): group-community
- Project Administrator (email): jcoghlan@gitlab.com 

### Provide a brief overview of the reason for this project and why it is needed and for how long it will be used.

Will primarily be used by the Developer Evangelism team in testing, demos, and content creation. 

## Security

### Provide a list of data and the corresponding classification that will be used in this project and how it will be accessed.


## Group Project Access Checklist

Make sure the following criteria is met and understood by the project administrator. 

- [ ] If the gitlab.com database is copied, that data has been processed by the [pseudonymization script]( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/pseudonymization-gitlab-db.md).
- [ ] Regular security updates are applied to all nodes in the project.
- [ ] Unused instances will be removed in a timely manner
- [ ] The Project Administrator is responsible for any users or additional administrators that they add to the project
- [ ] The Project Administrator is responsible for justifying any cloud spend within the project.
- [ ] Group Projects are intended for development, test, or demo work. Everything in these projects is considered temporary.

## Infrastructure Tasks

- [x] Create a new branch that is **not** the same as the group name and is less than 25 characters long. For example, `add-telemetry-group`.
- [x] Create file in https://ops.gitlab.net/gitlab-com/group-projects named `environments/(group name from above).tfvars` by copying an existing file and changing the Administrator and Group Name variables
- [x] Once the pipeline succeeds, review the changes are correct and stop the review by activating the `stop_review` job
- [x] Merge the change to master
- [x] Create a branch from master named `(group name from above)` and push
- [x] Verify that the pipeline completed successfully at https://ops.gitlab.net/gitlab-com/group-projects/pipelines
- [ ] (Optional) If the group does not start with `group-*` or `gitlab-qa-*k`, add the newly created branch as a protected branch.

Story points: 2.0
PoC migrating GET to collections format

As noted in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12787#note_542492346, shifting the ansible code within the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) and publishing the package to Galaxy would enable us to use that as an upstream dependency upon which we would build our gitlab.com-specific code. Ideally, when we begin shifting functionality from our existing Chef cookbooks, this would enable us to focus general updates and improvements to GET for all of its downstream users to benefit, and things which are highly specific to the gitlab.com deployment within GCP to a `gitlab.gitlab-com` collection.

This idea is not fully fleshed out, yet, and may not be feasible to incorporate with our current processes/requirements for managing the various roles/fleets within our infrastructure.

Story points: 5.0
Update CSP directives for embedded iframes

Many teams are creating curated learning paths/partner content in the EdCast platform (GitLab Learn) and ImPartner using SSOT content from the GitLab docs site and handbook. But if a user clicks the Web IDE button at the bottom of a handbooks page from within the iframe in either, the iframe goes blank.

We need to support embedded content from
```
edcast.gitlab.com
gitlab-learn.leapest.com
partners.gitlab.com
```

The definition of done here is to ensure that the 3rd party sites listed above where we're embedding content from the handbook are allowed in the CSP `frame-ancestors` directive within Fastly and the same for gitlab.com and Cloudflare (for the WebIDE). Ideally, we will also be able to test against staging before changing the production config.

For additional context, this was highlighted while [updating CSP directives for the handbook](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12062#note_541359829) to enable clickable SVG objects, and the [updated CSP inadvertently blocked embedded iframes](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/10052#note_541406634) from external 3rd party sites GitLab teams are using to provide educational content, partner resources, etc. This [has been an issue](https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/7638#note_341249056) in the past, as well.

Story points: 3.0
SRE Onboarding template assumes the engineer has SSH access for Chef steps

As an onboarding engineer, the steps to configure Chef (and knife) require that an SSH user is configured via the Chef data bags to allow SSH to the chef server. The order of these steps should be updated to reflect this dependency and a caveat should be added to the Chef steps for creating the Chef user so that it is understood that SSH steps will need to be completed first.

Story points: 1.0
Increase frequency of `ANALYZE` operation on `namespaces` table to every `30 minutes`

Increase frequency of `ANALYZE` operation on `namespaces` table to every `30 minutes`.

Alvaro Hernandex writes:

> Current incident started ~ 40mins after the last cron-ed ANALYZE. So probably 1h is not frequent enough. I'd suggest to run it more frequently, every 30 mins or even 15 minutes. It is a lightweight operation, which causes a bit of I/O and lasts, under weekend non-incident profile, just 18 seconds (probably under peak load on a week day more, but still quite lightweight).

@brentnewton writes:

> based on this, let’s move to every 30 mins.

~"corrective action" for production incident https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4024

Story points: 1.0
Replace CustomerDot Zuora integration User to Remediate SOX Gap

The CustomerDot Integration User is currently tied to a specific engineer's email address.  The following Issue contains additinoal context:  https://gitlab.com/gitlab-com/business-ops/enterprise-apps/intake/-/issues/91.  @jameslopez [has provided advice on what email should be associated with the CustomerDot integration user](https://gitlab.com/gitlab-com/business-ops/enterprise-apps/intake/-/issues/91#note_520357501):  `fulfillment-be+zuoraprod@gitlab.com`

The ask here is to work with the Infrastructure team to complete the switch of the CustomerDot integration user from `ruben_APIproduction@gitlab.com` to `fulfillment-be+zuoraprod@gitlab.com`

Story points: 3.0
Disable Cloudflare `Onion Routing` setting

**Current Situation**
<!--Describe the current situation which you are intending to iterate on with the work described in this issue.-->
The `Onion Routing` setting is enabled on GitLab.com. This causes Cloudflare to issue an `alt-svc` header to tor browsers.  
It is presumed, that the behaviour causes issues with Cloudflare Spectrum to trigger https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10860. This makes the site *unusable through tor* without disabling that header manually client-side.

![Screenshot_from_2021-03-12_18-40-01](/uploads/8daccc432b27ca23241f5caf267b0230/Screenshot_from_2021-03-12_18-40-01.png)

**Desired Outcome**
<!--Describe the intended outcome of the work represented in this issue. How does it improve upon the current situation above?-->
Disable the `Onion Routing` setting. That will cause TOR traffic to run through the usual tor relays and not be short-circuited to Cloudflare via a hidden service. Customers might see the Challenge page, but afterward, it should work normally, as we don't hit the issue in #10860.

This fix is easy to apply and at least will make the site usable, albeit with some inconvenience to the user.

https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/blob/cc7bfef7e45642aebaff8c19a8db936da21d361e/environments/gstg/main.tf#L2058 must be set to `off`  
https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/blob/cc7bfef7e45642aebaff8c19a8db936da21d361e/environments/gprd/main.tf#L2208 must be set to `off`

---
This might also be an upstream but in Cloudflare, as we use Cloudflare Spectrum for our HTTPS ingress. We should raise a case with them to see if that may contribute to the problem, as their hidden services might interfere with that/affect this behavior.
---

**Acceptance Criteria**
<!--List out work items necessary to arrive at the desired outcome above, as atomically as possible.-->
- [ ] `Onion Routing` is turned off on gstg
- [ ] `Onion Routing` is turned off on gprd

Story points: 1.0
Hook onto GCP instance maintenance events

During https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3891 we discovered, that we can react to imminent maintenance events by [hooking into a GCP API](https://cloud.google.com/compute/docs/storing-retrieving-metadata#maintenanceevents).

I wrote a little script that registers itself and listens for those events.

If everything is okay, we run all executable files in `/etc/gcp_watchdog/hooks/up`, if there is a problem/maintenance we run them in `/etc/gcp_watchdog/hooks/maint`.  
Every script will get the event (or `UP` for okay events) as argv1. Those can also be symlinks

`/dev/shm/gcp_watchdog.state` will contain the state/last event or `UP`.

If the metadata API is unavailable, a maintenance will be assumed and `metadata_unavailable_without_state` will be passed as the event. If there is a state saved in the current execution, that will be used instead.

Only a change in state will cause hooks to run.

We should seek to deploy this everywhere. We can then add some cheap scripts to see if it triggers as expected before hooking into production services.

```bash
#!/usr/bin/env bash

set -uf

STATE=/dev/shm/gcp_watchdog.state
HOOK_DIR=/etc/gcp_watchdog/hooks

# Clean the state. Others might consume this, but we want to always be in the know of what happens.
touch "${STATE}";
truncate -s0 "${STATE}"

handle_event () {
  local event="$1"
  local subdir="$2"

  if does_state_match "${event}"; then
    # We are already there.
    return 0;
  fi

  echo "Got new event: ${event}"

  # Find all executable hooks and execute 16 at a time, passing the event as argv1. OR-ing true to prevent xargs from qutting on error.
  find "${HOOK_DIR}/${subdir}" -executable -print0 | xargs -0rn1 -P16 -I% -- sh -c "% ${event} || true";

  set_state "${event}"
}

set_state () {
  local event="$1"

  echo "${event}" > "${STATE}"
}

does_state_match () {
  if [ "$(cat "${STATE}" 2>/dev/null)" == "$1" ]; then
    return 0;
  else
    return 1;
  fi
}

response="$(curl -s http://metadata.google.internal/computeMetadata/v1/instance/maintenance-event -H 'Metadata-Flavor: Google')"
while true; do
  if [ -z "${response}" ]; then
    # request failed
    echo request failed
    if [ -z "$(cat "${STATE}" 2>/dev/null)" ]; then
      echo "no state registered. Assuming maintenance mode"
      handle_event "metadata_unavailable_without_state" "maint"
    fi
    # Do not wait for changes, but fetch value ASAP
    response="$(curl -s http://metadata.google.internal/computeMetadata/v1/instance/maintenance-event -H 'Metadata-Flavor: Google')"
    sleep 1;
    continue;
  elif [ "NONE" == "${response}" ]; then
    # all clear, poll again
    handle_event "UP" "up"
  else
    handle_event "${response}" "maint";
  fi
  sleep 1;
  # For subsequent requests: Poll
  response="$(curl -s http://metadata.google.internal/computeMetadata/v1/instance/maintenance-event?wait_for_change=true -H 'Metadata-Flavor: Google')"
done
```

Story points: 2.0
Expose the container registry database-related metrics in the Grafana dashboards

Related to https://gitlab.com/groups/gitlab-org/-/epics/5392.

As we approach the deployment of the new registry with a metadata database and online garbage collection to pre-production, we'll need to extend the existing Grafana dashboards for the registry with database-related metrics (application metrics, not the database cluster metrics).

The container registry, when the metadata database is enabled, emits the following sets of Prometheus metrics:

- Connection pool statistics: Count of open, idle, in use, etc. connections ([`sql.Stats`](https://golang.org/pkg/database/sql/#DBStats));
- Statements duration: The duration of every single statement executed against the database is recorded using a histogram;
- Online GC statistics: Statistics about online GC, such as the number of processed tasks, recovered space, etc.

The ~"group::package" team will try to self-serve these changes but we may need some guidance.

Story points: 3.0
Rebuilt org-ci to increase IP address space

In the effort to roll out https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3648, we discovered there isn't enough IP space for doing an upgrade. We need to change the CIDR and thus rebuild the cluster.

The suggestion is to update `gitlab-gke-pod-cidr` to be `10.3.0.0/16`.

I estimate this won't take much time as there is tolerance for downtime with no customer impact as well as it being a pretty simple change. We will also need to be sure that the helm configs are pushed up to the new cluster.

We also need to update the [runbook](https://ops.gitlab.net/gitlab-com/runbooks/-/tree/master/docs/ci-runners#gitlab-org-ci-project) with new ranges.

Story points: 4.0