giseldo/neodataset huggingface dataset:
Use one consistent method to manage secrets buckets

We have added the `env-projects` environment to gitlab-com/gitlab-com-infrastructure> in order to start managing GCP projects and the service accounts used to provision resources into those projects via infrastructure-as-code (terraform) using the https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project module. As a follow-on step, we need to sort out how we handle [the secrets bucket](https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project/blob/8846b4b6ba45d1917ceaa1b33cc465b085a9d35c/main.tf#L29) - we have conflicting buckets created in multiple different modules, and the `project` module likely isn't the best place for them.

Some are created using the `project` module, while others are created using the `gitlab-storage` module, and we have various permutations of attributes in either case.

Story points: 2.0
Document bootstrapping GCP projects with terraform

We have added the `env-projects` environment to gitlab-com/gitlab-com-infrastructure> in order to start managing GCP projects and the service accounts used to provision resources into those projects via infrastructure-as-code (terraform) using the https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project module. As a follow-on step, we need to add/update documentation for project bootstrap and management process (especially how to handle circular dependency to manage `env-zero` in terraform (if possible), as well)

Story points: 1.0
Add IAM permissions for terraform service accounts

We have added the `env-projects` environment to gitlab-com/gitlab-com-infrastructure> in order to start managing GCP projects and the service accounts used to provision resources into those projects via infrastructure-as-code (terraform) using the https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project module. As a follow-on step, we need to add/import/update the relevant IAM permissions for those service accounts.

Story points: 1.0
Manage the registry application grafana dashboards with libsonnet

* https://dashboards.gitlab.net/d/CoBSgj8iz/application-info?orgId=1
* https://dashboards.gitlab.net/d/oWe9aYxmk/pod-metrics?orgId=1

These need to be converted over from the current manual creation method into libsonnet.


/cc @gitlab-com/gl-infra

Story points: 5.0
Why do the PgBouncer hosts have a very uneven distribution of client connections?

**Problem:**

A large majority of database client connections are being handled by a single PgBouncer instance, despite the fact that 2 PgBouncer instances are active backends in our Google Internal TCP Load Balancer.  This imbalance defeats our attempt to alleviate our bottleneck of PgBouncer's CPU saturation.

**Goal:**

Learn why traffic is not being roughly equally distributed among the active members of the Google Internal TCP Load Balancer (ILB).  This is a research task that should result in a recommendation.

**Alternatives:**

If we cannot fix this imbalance, we could instead use HAProxy as a load balancer in front of several PgBouncer instances.  This would optionally let us run multiple PgBouncer instances per host.  However, unlike the Google ILB, HAProxy would be an additional TCP endpoint, adding more network latency and another component to manage.

**Background:**

 - Recently (~4-8 weeks ago) in response to a performance problem we changed how database clients connect to the primary Postgres instance in our Patroni cluster.
   - Previously, all clients needing write-access to Postgres would connect to a single PgBouncer instance running on the same host as the primary Postgres instance.
   - PgBouncer is a single-threaded process, so it can use at most 1 CPU's worth of compute cycles.
   - That PgBouncer instance's workload sometimes saturated its 1 CPU ceiling under our peak daily workload.
   - To spread that workload among more CPUs, we needed to add more PgBouncer instances and treat them as a pool.
 - We reconfigured the DB Clients connect to the primary Postgres database via a Google Internal TCP Load Balancer (ILB) that points to a pool of currently 2 active PgBouncer instances.
   - The ILB is not a proxy; it is a set of network routing rules that lets clients talk directly to the backend service instances (PgBouncer hosts).
   - The ILB's backend-service actually contains 2 active hosts and 1 inactive host.  Only the 2 active hosts matter for this analysis.
 - Surprisingly, the workload is *not* evenly distributed among the PgBouncer hosts.
   - The number of established connections from clients to PgBouncer is very unevenly distributed among the 2 active PgBouncer hosts.
   - This unevenness defeats our goal of spreading the workload among multiple PgBouncer instances.
   - Clients whose hostnames start with "web" and "git" seem to be the most strongly affected by this bias for preferring to connect to host "pgbouncer-03" rather than "pgbouncer-02".

Story points: 5.0
Update DBRE section of On-Call Handbook section

The [DBRE section](https://about.gitlab.com/handbook/on-call/#dbre) of the On-Call Handbook reads:

> For database-related issues the DBRE on-call can be paged.
> Responding to pages is on a best-effort basis and there is no expected response time.
> Alerts are not triggered via automation. All escalations to the DBREs are initiated by a human, either the SRE or manager on-call.

This needs to be updated, given that we do have an expected response time. Also, I believe we do have automated alerts.

Story points: 2.0
FY20-Q3 Infrastructure IACV OKR: Raise GitLab.com Availability from 96.64% to a consistent 99.95% => 0%

* [ ] Key result: MTTD, MTTR, MTBF, and MTTP PIs to be at OK health and a minimum of level 2 maturity `=> 0%, comment`
* [ ] Key result: 100% of services have fully defined and observable service levels, error budgets, and capacity planning `=> 0%, comment`
* [ ] Key result: 50% of services running on Kubernetes `=> 0%, comment`

Story points: 8.0
FY20-Q3 CI/CD & Enablement: Drive all user-visible services’ MTBF from 4 days to 6 days => 0%

Current MTBF calculations spreadsheet: https://docs.google.com/spreadsheets/d/1bCEUQbWMccVm1dfBrOFtXjyDVCflHu4rB6htOaC_0yI/edit#gid=1150439504

4 days to 6 days is based on calculation of MTBF for S1/S2 incidents

Story points: 1.0
FY20-Q3 OKR: Drive all user-visible services’ MTTD from Unknown to 5 min => 0%

* [ ] Key result: 70% (7/10) DevOps lifecycle stages have SLI dashboards for development to monitor. `=> 0%, comment`
* [ ] Key result: 100% of team-owned services have fully defined and observable service levels, error budgets, and capacity planning `=> 0%, comment`
* [ ] Key result: 50% of team-owned services running on Kubernetes `=> 0%, comment`
* [ ] Key result: Perform weekly load-testing in staging on 2 team-owned services `=> 0%, comment`

Story points: 8.0
FY20-Q3 Dev & Ops OKR: Drive all user-visible services’ MTTR from 4.66 to 2 (days) => 30%

* [ ] Key result: 100% of incidents have severities, RCAs, and error budgets accounted for `=> 0%, comment`
* [ ] Key result: 100% of team-owned services have fully defined and observable service levels, error budgets, and capacity planning `=> 0%, comment`
* [ ] Key result: Migrate Postgresql and his ecosystem to use stackgres( kubernetes) in staging. `=> 0%, comment`

Story points: 8.0
Error when deploying customers.gitlab.com

```
    - gitlab-mitigate-sackpanic (0.1.3)
  - gitlab_users (0.1.49)
  - chef-client (11.2.0)
  - gitlab-prometheus (1.2.0)
  - systemd (3.2.4)
  - windows (6.0.0)
  - gitlab-server (1.1.0)
  - gitlab-exporters (1.3.1)

Running handlers:
[2019-08-02T20:07:32+00:00] ERROR: Running exception handlers
Running handlers complete
[2019-08-02T20:07:32+00:00] ERROR: Exception handlers complete
Chef Client failed. 0 resources updated in 11 seconds
[2019-08-02T20:07:32+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2019-08-02T20:07:32+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-08-02T20:07:32+00:00] FATAL: Chef::Exceptions::CookbookChefVersionMismatch: Cookbook 'gitlab_application_ruby' version '4.2.3' depends on chef version [">= 12.1", "< 14"], but the running chef version is 14.13.11
```

Story points: 1.0
We need a way to test kubernetes application configuration changes locally

Right now testing application changes must happen on an environment, whether that be `gstg` or `pre`.  This is not the safest way to test things if multiple persons are testing.  It would be nice to utilize Docker's Kubernetes integration locally, or minikube, or another tool to test application configuration changes as necessary.

@jarv recently demo'd the use of docker-desktop

@skarbek has toyed with minikube

We need to ensure that whatever tool we utilize, it's backward compatible with linux and osx.

Use this issue to track progress on making this reality.

Story points: 5.0
The dbconsole-praefect.sh script must specify secrets for database client connections

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details**
 - Point of contact for this request: @nnelson
 - If a call is needed, what is the proposed date and time of the call:  [+ Date and Time +]
 - Additional call details (format, type of call): [+ additional details +]

**SRE Support Needed**

The `dbconsole-praefect.sh` script is now required to specify the client certificates to be used for the connection.

This may be verified with:

```sh
ssh console-01-sv-gstg.c.gitlab-staging-1.internal
PGSSLCOMPRESSION=0 PGPASSWORD=$(sudo /bin/cat /etc/gitlab/gitlab.rb | perl -ne 'm/praefect.*database_password.*?"(.*)"/ && do { print $1 }') sudo --preserve-env /opt/gitlab/embedded/bin/psql "sslmode=verify-ca sslrootcert=/etc/gitlab/ssl/praefect-database-server-ca.pem sslcert=/etc/gitlab/ssl/praefect-database-client-cert.pem sslkey=/etc/gitlab/ssl/praefect-database-client-key.pem hostaddr=10.94.0.2 user=praefect dbname=praefect_production"
```

The template and recipe in the unfortunately scoped `gitlab_users` cookbook must be updated accordingly.


<!--
please do not edit the below
-->

Story points: 4.0
Container Registry: Review and adjust SLI for manifest routes

## Context

Related to https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/476+.

## Task

Now that Phase 2 of the [GitLab.com registry upgrade/migration](https://gitlab.com/groups/gitlab-org/-/epics/5523) is complete, the existing SLIs are likely too loose, as operations are now executed on the metadata database (faster), not the storage backend (slower).

~"group::container registry" will self-serve this issue.

Story points: 1.0
Establish a method for monitoring the health/success of disk backup snapshot creation in GCP

## Summary

<!--
Give context for what problem this issue is trying to prevent from happening again.

Provide a brief assessment of the risk (chance and impact) of the problem that this corrective action fixes, to assist with triage and prioritization.
-->

Establish a method for monitoring the health/success of disk backup snapshot creation in GCP.

There is presently a now known risk that disk backup snapshots could begin to fail without anyone being aware of the situation until one goes to try to use a snapshot which has failed for something.

Failed disk backup snapshots could result in the extensive loss of customer data were any of the virtual server disks to fail for any reason during a period of failing snapshot creation.


## Related Incident(s)

<!--
Note the originating incident(s) and link known related incidents/other issues.

The relation will happen automatically if you are creating this issue from an incident, if this isn't done already please uncomment the following line:
-->

Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7833


## Desired Outcome/Acceptance Criteria

<!--
How will you know that this issue is complete?

If you have any initial thoughts on implementation details (e.g. what to do or not do, gotchas, edge cases etc.), please share them while they are fresh in your mind.
-->

An alert will be generated, and ultimately the SRE On-call will be paged, when a certain number (any?) of disk backup snapshots fail to be created within a certain period of time from their scheduled initialization.


## Associated Services

<!--
Apply the appropriate services associated with this corrective action if applicable.

~Service::SERVICE_NAME

-->

~"Service::GCP" 

## Corrective Action Issue Checklist

* [x] Link the incident(s) this corrective action arose out of
* [x] Give context for what problem this corrective action is trying to prevent from re-occurring
* [x] Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
* [x] Assign a priority (this will default to 'Reliability::P4')

Story points: 3.0
Demo: /chatops run pager pause

# Demo: `/chatops run pager pause`

## Scenario

> For site-wide outages, EOC gets a nearly continuous stream of pages that needs to be acknowledged, which is very distracting when trying to focus on mitigating said outage. For production, we have 3 PagerDuty services that could send a page to the EOC: `GitLab Production`, `SLO Alerts gprd main stage`, and `SLO alerts gprd cny stage`. Manually creating a maintenance window for all of them is a waste of time during an incident.

See this ~"corrective action" for details: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15550

Moderator note: Since Steve Azzopardi's excellent work creating relationships and hierarchies for service alerting, this has since become much less of a burdensome problem during severe incidents. Nevertheless, this feature may have some value, and because it directly effects the way that an on-call engineer is paged or not, it is important to socialize this operational information.

See runbook documentation here: https://gitlab.com/gitlab-com/runbooks/-/blob/master/on-call/checklists/eoc.md#creating-temporary-pagerduty-maintenance-windows

## Meeting Format

Demo presentation.

- [x] Moderator: @nnelson 
- [ ] Note Taker: @shimrangeorge / `tbd`


## Acceptance Criteria

<!--
  All items must be completed prior to the closure of this issue.
-->

- [ ] Google Doc created: `todo`
- [ ] Meeting scheduled; Agenda should include
  - Google Doc
  - Link to Scenario
- [ ] Meeting must be recorded
- [ ] Recording is uploaded to YouTube; apply the video to the following
  playlists:
  - Infrastructure Fire Drills
  - Infrastructure Group
- [ ] Mark the video as private if any [Yellow and above classified data is
  shared](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html)
- [ ] Review the Google Doc and/or the Video for any potential follow up issues
  that need to be resolved

Story points: 1.0
For project authorizations refresh jobs, use workers based on the applicable urgency context

Based on the discussion at https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16080#note_1054295096

> granting access (inserting into `project_authorizations`) can have lower priority. According to https://docs.gitlab.com/ee/development/sidekiq/worker_attributes.html#job-urgency, this would mean a delay of maximum 1 minute before the users can actually access the project they've gained access to. But when revoking access, this might be to block a bad actor from something, so we might want it more immediate.

This means that we have an opportunity to have different types of workers that do the same thing, but differ in urgency.

And based on its context (eg, addition of a member to a project vs removal of a member from a project), we can use the worker based on the urgency of the user action.

Story points: 3.0
Decide tagging pattern for premium machine types

Premium machine types (such as n2-standard-4 to pick a totally arbitrary example) need some sort of standard tagging format to be determined ahead of time.

This small detail is important as it's very painful and annoying to users if we change it after the fact.

Premium machine type runner managers will be set to `run-untagged=false` so that the only way to use them is to run with the tags. 

Blocks #16091

## PM proposal for configuration and tagging pattern (2022-07-29)

### Tagging naming convention for GitLab SaaS runners

saas-{os}-{instance size}-{arch}-{additional capability}

- SaaS: denotes that this is a SaaS Runner
- OS: denotes the operating system.
- instance size: Using AWS's [naming convention](https://www.archerimagine.com/articles/aws/aws-ec2-instance-type-tutorial.html#instance-sizing) as prior art, this denotes the T-shirt size representation of the instance (CPU, Memory)
- additional capability: used to represent additional capability. For example - GPU enabled VM's.

| Description| Cloud Provider |Machine Type|Run Untagged Jobs|Tags|
| ------ | ------ | ------ | ------ | ------ |
| Linux + Docker Builds - [general purpose CI] | Google Compute |n1-standard-1|Yes|saas-linux-small-amd64|
| Linux + Docker Builds - [general purpose CI] | Google Compute |~~n1-standard-2~~ <br> n2d-standard-2|No|saas-linux-medium-amd64|
| Linux + Docker Builds - [general purpose CI] | Google Compute |~~n1-standard-4~~ <br> n2d-standard-4|No|saas-linux-large-amd64|
| Linux + Docker Builds - [AI/MLOPS - GPU Enabled] | Google Compute |n1-standard-4 + GPU|No|saas-linux-large-amd64-gpu|

Story points: 0.0
Create terraform plans for large machine runner managers

Add new runner managers into terraform repository.

## Work checklist

- [x] merge https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4139
- [x] request quota limit upgrade of C2 CPUs to 2000
- [x] apply changes from https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4139 to create the runner manager nodes
- [x] apply fix from https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4155
- [x] manually verify configuration
- [x] finalize preparation steps (create Docker Machine VM on each of the nodes manually - once per each node - to make sure that the configuration works and to generate Docker Machine TLS Auth certificates)

Story points: 1.0
Register large machine runner managers

We need to register the new machines and store the tokens generated in our 1password vault.

To truly be done this, we need to know the tagging format to be used to distinguish the machine types. Tags can be updated after registration though so we can register, store the tokens, then update the tag formats.

**This issue requires gitlab.com admin access**

## Plan checklist

- Register following runner managers and store tokens in chef vault
  - `saas-linux-medium` :point_right: `./bin/gkms-vault-create runners-manager-saas-linux-medium-amd64 ci`
    - [x] `blue-1.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `blue-2.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `blue-3.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `blue-4.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `blue-5.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `green-1.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `green-2.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `green-3.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `green-4.saas-linux-medium-amd4.runners-manager.gitlab.com`
    - [x] `green-5.saas-linux-medium-amd4.runners-manager.gitlab.com`
  - `saas-linux-large` :point_right: `./bin/gkms-vault-create runners-manager-saas-linux-large-amd64 ci`
    - [x] `blue-1.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `blue-2.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `blue-3.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `blue-4.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `blue-5.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `green-1.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `green-2.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `green-3.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `green-4.saas-linux-large-amd4.runners-manager.gitlab.com`
    - [x] `green-5.saas-linux-large-amd4.runners-manager.gitlab.com`
- Add collected tokens to chef vault for the dedicated roles

Story points: 0.0
Create new GCP Projects for new Runner managers

As per https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/768#note_1066322541 we will define two new shards of our SaaS Linux runners: `saas-linux-medium` and `saas-linux-large`.

For each of them we will have 5 working runners, which means 5 GCP projects per shard. We need to register unique CIDRs for the `ephemera-runners/ephemeral-runners` subnetworks that will be used by the ephemeral VMs. Later, basing on the created projects, we will create runner managers connected with them.

## Plan checklist

- [x] Register unique CIDRs for ephemeral runner projects in our documentation :point_right: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4915/diffs
- [x] Define new projects through terraform :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4100
- [x] Create new projects manually :point_right: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7652
- [x] ~~Merge new environments definition :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4121~~
- [x] Configure the new projects through terraform :point_right: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4101 (please rebase after merging 4121!)

Story points: 4.0
Update home page url in staging from https://about.staging.gitlab.com to https://staging.gitlab.com/users/sign_in

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

**Details**
 - Point of contact for this request: @nnelson 
 - If a call is needed, what is the proposed date and time of the call:  `2022-07-22 1500 utc`
 - Additional call details (format, type of call): `No call required`

**SRE Support Needed**

Update home page url in staging from `https://about.staging.gitlab.com` to `https://staging.gitlab.com/users/sign_in`.

Navigate to [`https://staging.gitlab.com/admin/application_settings/general`](https://staging.gitlab.com/admin/application_settings/general) `> ` `Sign-in restrictions`

Change this field:

![Screen_Shot_2022-07-22_at_9.20.34_AM](/uploads/c3be969bc84070782f54b984a79139de/Screen_Shot_2022-07-22_at_9.20.34_AM.png)

To the new value: https://staging.gitlab.com/users/sign_in

Click `Save changes`.

<!--
please do not edit the below
-->

Story points: 1.0
Bring node_count from 2 to 3 for camoproxy service cluster in staging

Bring node_count from 2 to 3 for camoproxy service cluster in staging.

Part of: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/15277

For more details see: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/15277#note_852825341

Story points: 1.0
Increase machine type for staging camoproxy

Increase machine type for `staging` `camoproxy`.

So, a configuration for `osqueryd` appears to be causing leading to performance issues on small instances.

This issue tracks the work for increasting the machine type for the ~"Service::Camoproxy" instances in the `staging` environment.

Part of: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/15110

Story points: 1.0