giseldo/neodataset huggingface dataset:
DNS Record modification for domain gitlab.design

UX would like to run research using UX prototypes POCs and we need a domain name to host that under. We already have `gitlab.design` domain that is used to redirect to the Pyjamas site.

Can we have subdomain `research.gitlab.design` allocated for the prototypes? It would be great to add this as a DNS Zone in GCP, so that going forward, people can add second-level subdomains to this sub-domain and point their instances to them.

PS: The redirect site does not have TLS, we should fix that too while we're at it.

Story points: 1.0
Add Slack Integration to Sentry

**Problem:** Server 500 errors are being collected by Sentry and the appropriate teams are not learning about those errors on a timely basis. 

**Solution:**
I would like to configure Sentry to have richer Slack integration. This can be done following these steps:
https://forum.sentry.io/t/how-to-configure-slack-in-your-on-prem-sentry/3463

I have already created a Slack Application as per the configuration documents. 

The 3 pieces of data to complete the integration are 

```
slack.client-id: <'client id'>
slack.client-secret: <client secret>
slack.verification-token: <verification token>
```

These values need to be added to the config.yml for Sentry via the chef recipe. 

The values for those variables are stored in 1Password Team Vault under the record 'Slack App for Secure Sentry'.

Once those values are added to config.yml for Sentry and Sentry has been restarted, the "+Add Workspace" on this page should work, and allow us to configure the rest of the alerting rules. 
https://sentry.gitlab.net/settings/gitlab/gitlabcom/integrations/slack/

Story points: 1.0
Kubernetes ingress crashing for version.gitlab.com

The new kubernetes ingress for GitLab Services which is managed by the GitLab Kubernetes integration keeps crashing.  It's unclear whether this is a problem with the integration, with the app, or with the version of kubernetes that we are running.

The migration from AWS to GKE has been rolled back until this is solved.

The ingress pods show `0 of 1 updated replicas available - Pods have warnings`

And the Pod Logs show:

```
I1018 02:53:08.287037       7 main.go:150] Received SIGTERM, shutting down
I1018 02:53:08.287102       7 nginx.go:321] shutting down controller queues
I1018 02:53:08.287131       7 status.go:115] updating status of Ingress rules (remove)
I1018 02:53:08.303618       7 status.go:134] removing address from ingress status ([35.243.233.112])
I1018 02:53:08.308592       7 status.go:342] updating Ingress version-gitlab-com-6491770-production/production-auto-deploy status to []
I1018 02:53:08.316522       7 nginx.go:329] stopping NGINX process...
```

The ingress in question is: https://console.cloud.google.com/kubernetes/deployment/us-east1/gs-production-gke/gitlab-managed-apps/ingress-nginx-ingress-controller?project=gs-production-efd5e8&authuser=1&cloudshell=false&organizationId=769164969568&tab=overview&deployment_overview_active_revisions_tablesize=50&duration=PT1H&pod_summary_list_tablesize=20&service_list_datatablesize=20


//cc @jameslopez

Story points: 2.0
Investigate intermittent response time spikes concurrently affecting multiple services

Hunt for clues about intermittent slowness reported for GitLab.com.

During at least the last 2 days (maybe more), several internal users have reported observing brief periods of slow performance on the web UI and git-push.  These reports so far lack details, but reviewing the frontend metrics revealed several brief but large spikes in the 50th percentile response times recorded by HAProxy.  Several (but not all) of HAProxy's backend service pools concurrently reported these spikes, which implies a shared dependency of these specific services may have been stressed.

This issue aims to:
* Document the findings so far.
* Try to identify which internal services were stressed during a few of these spikes.
* If a pattern emerges, try to determine more about the nature of the bottleneck (e.g. higher overall request rate, higher rate of slow/costly requests, contention over a machine resource, lock contention, saturation of a resource pool, etc.).

Story points: 2.0
Scale up `review-apps` runners for about-src.gitlab.com

Right now we have 80+ jobs stuck waiting for the single `about-src.gitlab.com` runner to pick up the `review-apps` jobs:

![image](/uploads/4e991349df60e5e7b18b010587e2207c/image.png)

https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs?page=2&scope=pending

I think we should scale this up, although the bulk of the time may be downloading the artifacts. We may need more cores on that machine as well to handle this.

Story points: 6.0
Investigate running Discourse on GKE

We're looking to [migrate forum.gitlab.com to GKE](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/101). This issue is to track research on how we're going to be able to do that. 

Test project: https://gitlab.com/cody/discourse-k8s-sandbox-2

## Can we use AutoDevOps?

I think we can probably use AutoDevOps for the build and deployment of Discourse. But I'm still investigating how that might work.

Story points: 5.0
Enable HTTP compression on customers.gitlab.com

# Summary
https://customers.gitlab.com doesn't have HTTP compression enabled. See [this report](https://developers.google.com/speed/pagespeed/insights/?url=http%3A%2F%2Fcustomers.gitlab.com%2F) from PageSpeed.

# Proposal
Enable gzip compression on nginx. This will greatly improve performance and shouldn't cause any issues.

Story points: 1.0
Request access to ops.gitlab.net and restore project for OnGres team

We need access to https://ops.gitlab.net and https://dashboards.gitlab.net/ for all the @gl-consultants-ongres team, in order to move forward with some issues. 

We already have access to the restore project (https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore). 
 
cc/ @glopezfernandez

Story points: 2.0
Multiple configuration issues with Prometheus in Kubernetes

* [x] Failing rule evaluations are showing up in the logs quite often
* [x] No monitoring of Prometheuses running in our clusters (such as the ops environment monitoring all our existing prometheus servers)
  * We are configured to have at least 2 Prometheus Pods running in `gprd` but only 1 is running currently: `up{cluster="gprd-gitlab-gke",env="gprd",job="gitlab-monitoring-promethe-operator"}` and we are completely unaware of this missing redundancy
* [x] No external mechanism to reach the prometheus endpoint in our clusters
  * We only expose the thanos endpoint.  One must utilize `kube port-forward` to reach the prometheus endpoint: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/monitoring
* [x] Scraping data appears to be inconsistent 
  * Metric scrapes in Kubernetes appear to only be capturing one metric over the course of time, example query: `kube_pod_container_info{container="registry", cluster="gprd-gitlab-gke"}[5m]`
* [x] Deduplication of mulitple prometheus instances is not occuring
  * Example query: `count (kube_pod_container_info{container="registry", cluster="gprd-gitlab-gke"}) without(cluster,container,container_id,endpoint,env,environment,image,image_id,instance,job,namespace,pod,prometheus,provider,region,service)`
  * Returns multiple data points: 
```
✔{prometheus_replica="prometheus-gitlab-monitoring-promethe-prometheus-1"}
✔{prometheus_replica="prometheus-gitlab-monitoring-promethe-prometheus-0"}
```

All of the above is contributing to awkward metrics at times.  Specifically, noting our Container Registry dashboard: https://dashboards.gitlab.net/d/8wlZHpTZz/registry-pod-info-copy?orgId=1&from=now-3h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-cluster=gprd-gitlab-gke&var-namespace=monitoring&var-Node=All&var-Deployment=gitlab-registry, some metrics appear duplicated at times, which is impacting our ability to properly monitor our clusters with confidence.


/cc @ansdval

Story points: 8.0
file-33 and file-34 need to be rebalanced

The gitaly nodes file-33 and file-34 have crossed the 80% disk usage threshold and need to have projects moved to less full servers. For reference: https://gitlab.com/gitlab-com/runbooks/blob/master/howto/sharding.md

@dawsmith I don't know if this is SRE-Oncall or part of normal workflows for one of the teams.

Story points: 2.0
Move terraform GCS instance service account to project module

While cleaning up terraform code for `gitlab-ENV-secrets` buckets duplicated between the `project` and `storage-buckets` modules, [this block](https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/storage-buckets/blob/master/main.tf#L73-81) was leftover creating the IAM binding granting access to the `terraform` service account used for all GCE VM instances.

With the secrets bucket moved to the project module, it doesn't make sense to keep this IAM binding in the old module, but moving it implicates also managing the `terraform` service account with the project, and exporting the account email/ID for other terraform configurations to access via remote state lookup for the bindings that _are_ staying in the `storage-buckets` module. I inspected the terraform state files for `gstg` and `gprd` and it appears that this service account is not currently managed by terraform, and so can easily be imported into the `env-projects` configuration after the `project` module is updated.

1. [ ] Add a `google_service_account` resource to the `project` module for the `terraform` account.
1. [ ] Remove the `google_storage_bucket_iam_binding` from the `storage-buckets` module
1. [ ] Add a `google_storage_bucket_iam_binding` to the project module
1. [ ] Import the service account to the terraform state for `env-projects`
1. [ ] Import the `google_storage_bucket_iam_binding` resource to the terraform state for `env-projects`
1. [ ] Bump module versions for `project` and `storage-buckets` across all environments

Story points: 3.0
Setup logging for Praefect on gstg

As part of its [readiness review](https://gitlab.com/gitlab-com/gl-infra/readiness/merge_requests/10), we must setup logging for Praefect. Since we already deployed it on gstg we can setup its logging infrastructure there so we can verify it and replicate it later on gstg. We'll need:

- [x] A pubsub host: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1103
- [x] A gitlab_fluentd recipe: https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/merge_requests/92
- [x] Add the recipe to our praefect hosts: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1983

Story points: 3.0
Services resource deprecated for google_project_services

The `google_project_services` resource used in the project module is deprecated.

```
Warning: google_project_services is deprecated - many users reported issues with dependent services that were not resolvable.  Please use google_project_service or the https://github.com/terraform-google-modules/terraform-google-project-factory/tree/master/modules/project_services module.  This resource will be removed in version 3.0.0.
```

Story points: 1.0
Synthetic system halt test in staging

The `file-33-stor-gprd` project storage node is currently experiencing repeated unexplained system halts.

In order to support migrations for re-balancing large projects off of `file-33-stor-gprd` and onto `file-40-stor-gprd`, it is important to collect some data on what to expect if a source system were to halt in the middle of a project migration.

Planning the steps to undertake in orer to accomplish a test of this scenario, as well as the documenting of the consequences of the expected failures, will be the focus of this issue.

## Plan

1. [x] Create a new file storage node.
1. [ ] Enable the node to be used as storage using the GitLab admin console.
1. [ ] Create or move a handful (at least three or four, maybe more) projects
1. [ ] Begin migration of project from `file-04-stor-gstg` to one of the other nodes.
1. [ ] Meanwhile, in a shell session in the `file-04-stor-gstg` system, invoke the following commands: `sudo sysctl -w kernel.panic=10; sudo sysctl -w kernel.sysrq=1; echo c | sudo tee /proc/sysrq-trigger`
1. [ ] Now observe the serial port logs for `file-04-stor-gstg` at https://console.cloud.google.com/compute/instancesDetail/zones/us-east1-c/instances/file-04-stor-gstg/console?project=gitlab-staging-1&organizationId=769164969568#end
1. [ ] Analyze the migration logs and the system logs to determine the consequences of the system halt on the project migration.

Story points: 5.0
Add metadata to terraform managed projects

Some of our GCP projects are managed by the [Project](https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/project) terraform module.  Some are managed manually.  There is currently not an easy way to tell which is which without asking someone or checking many different repositories.

I propose altering the Project module to add some metadata to all projects indicating that it is managed by terraform and should not be manually changed.  This could be as simple as adding a `terraform-managed` tag like we have on some other resources.

/cc @Craig @cmcfarland

Story points: 1.0
file-41 was not provisioned properly

During this incident https://gitlab.com/gitlab-com/gl-infra/production/issues/1222 we created file-41 and ended up not using it, but kept it anyway as it would have to be created at some point. There were issues with initial provisioning. We've hit this https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6933 , manual rerun of `chef-client` was successful. Then, gitaly was complaining that the `/var/opt/gitlab/git-data/repositories` directory was missing and indeed it was not there. The directory was actually up one level higher, and `/var/opt/gitlab/git-data` was missing. It was manually created, but gitaly still complained until `/var/opt/gitlab/git-data/repositories` was also created.

What remains to be done is:
- verify that file-41 is fully operational
- investigate why the directory was not created in the first place (find it in Chef code for example)
- investigate https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6933


/cc @craig

Story points: 3.0
Fix terraform CI permissions

While attempting to apply https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1095, the job returned the following errors:

```hcl
Error: Error creating Disk: googleapi: Error 403: Required 'compute.disks.create' permission for 'projects/gitlab-production/zones/us-east1-b/disks/api-21-sv-gprd-log'
More details:
Reason: forbidden, Message: Required 'compute.disks.create' permission for 'projects/gitlab-production/zones/us-east1-b/disks/api-21-sv-gprd-log'
Reason: forbidden, Message: Required 'compute.disks.setLabels' permission for 'projects/gitlab-production/zones/us-east1-b/disks/api-21-sv-gprd-log'


  on .terraform/modules/api/instance.tf line 152, in resource "google_compute_disk" "log_disk":
 152: resource "google_compute_disk" "log_disk" {



Error: Error creating Disk: googleapi: Error 403: Required 'compute.disks.create' permission for 'projects/gitlab-production/zones/us-east1-c/disks/api-22-sv-gprd-log'
More details:
Reason: forbidden, Message: Required 'compute.disks.create' permission for 'projects/gitlab-production/zones/us-east1-c/disks/api-22-sv-gprd-log'
Reason: forbidden, Message: Required 'compute.disks.setLabels' permission for 'projects/gitlab-production/zones/us-east1-c/disks/api-22-sv-gprd-log'


  on .terraform/modules/api/instance.tf line 152, in resource "google_compute_disk" "log_disk":
 152: resource "google_compute_disk" "log_disk" {
```

While I manually applied this change to resolve the error in the near term, we need to update the permissions assigned to the `terraform-ci` service account, to assign the `compute.disks.create` permission as noted for the long-term fix.

/cc @gitlab-com/gl-infra/secure-and-defend

Story points: 1.0
Plan to synthesize a system halt on staging storage node `file-02-stor-gstg.c.gitlab-staging-1.internal` during a project migration

The `file-33-stor-gprd` node usage has exceeded our SLO target.  However, it continues to experience unexpected system halts on a nearly daily basis.

In order to determine the level of severity of and to otherwise understand the consequences of a system halt while in the midst of the procedure to copy a file from one storage node to another, it is necessary to design the replication of such a scenario in our staging environment.

Story points: 4.0
Investigate Exporting Stackdriver Metrics for kernel panics and reboots to Kibana or Prometheus

## Problem Summary

#7872 is currently being used as a comment collector for tracking reboots after manually parsing through the StackDriver logs. Rather than checking this daily, let's automate the collection of data points by either counting the logs in Kibana or using an exporter to aggregate in Prometheus. I have no strong preference for Kibana or Prometheus, but depending on the log format I expect one may prove to be easier to accomplish than the other. If that's true, we should choose the path of least effort.

### Definition of Done

This issue is primarily focused with determining how this log forwarding or metrics aggregation can be achieved, if at all. The work to develop the query and write a runbook pointing to the dashboard is in scope, but not required to close this issue. If writing the runbook and creating the dashboard is achievable with minimal effort, it can be tracked her. Otherwise, a new issue should be opened and linked here.

Story points: 2.0
Document Patroni, PGBouncer, and Consul Design and Troubleshooting in Runbooks

Discovered #8050 after opening this ticket. They're now linked.
<hr>
Per https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5844#note_223351269, we need to update our [Runbooks](https://gitlab.com/gitlab-com/runbooks) to include more information about how Patroni relies on Consul for health checks and leader election, and specific steps to diagnose and troubleshoot issues.

A good bit of that information is already captured in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7790. And will change after https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8050 is completed.

Below are more ~Discussion and ~Investigation issues @msmiley has authored that may or may not be valuable to add to the `howto` or `troubleshooting` sections in the Runbooks project repository.

- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7440
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7735
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7813
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7551

Story points: 3.0
gitlab-iptables appends new rules after final drop rule

When adding new sources to an existing firewall rule attribute in the form of

```
"firewall": {
      "rules": [
        {
          "Prometheus Server - Prometheus server internal metrics access": {
            "port": "1234",
            "protocol": "tcp",
            "source": "1.2.3.4,5.6.7.8"
          }
        },
        ...
```

then the `gitlab-iptables` cookbook is appending iptables rules after the existing rules including the `-j DROP` match-all target which is stopping the evaluation of newly appended rules. To fix it, the `/etc/iptables.d/` directory probably needs to be cleaned from existing rules before changes are applied.

Story points: 3.0
Provision infrastructure for new chef server(s) in gitlab-ops project

In preparation for the [migration to a new, upgraded Chef server](#6128), we need to build the infrastructure on which we will deploy Chef server. ~~For resiliency purposes, we should implement a [HA architecture](https://docs.chef.io/install_server_ha.html)~~

In light of the fact that we are focusing all new services to run in Kubernetes (barring exceptional circumstances), and actively working to migrate existing services away from GCE VMs, expending effort to build/manage a more complex configuration likely isn't worthwhile. Especially considering that historically we have experienced few (if any) scaling issues with a single Chef server node, as it stands.

1. [x] Create/deploy terraform modules and resources for chef infrastructure (load-balancer, ~~frontend instance group~~, backend instance group for self-healing configuration)
1. [ ] ~~Update monitoring for new Chef infrastructure (may make sense to split this into a separate issue)~~
1. [ ] ~~Update documentation in handbook (network/architecture diagrams) and runbooks to reflect the changes~~

Story points: 5.0
Identify updates required to chef-repo pipelines and related workflows

While working through the [Chef server migration and upgrade](#6128), we need to consider the impact to existing pipelines and related workflows around `chef-repo`. Specifically, do we need to make any syntactic changes to the scripts/jobs for the cookbook uploader, cookbook version bumps, role/environment pushes, and/or how we manage data bags and encrypted vault items. In addition, we need to ensure that during the course of the migration, while we're testing the new server and updated infrastructure, we push net-new changes to _both_ Chef servers, so that they stay up to date until we perform the final migration and cutover.

1. [ ] Review `chef-repo` pipeline for any syntax changes required by Chef server (latest as of issue creation is `13.0.17`)
1. [ ] Review runbooks and hoot documentation for references to workflows that may be impacted by the upgrade (knife commands, data bag/vault management scripts, etc.)
1. [ ] Update `chef-repo` pipelines to keep both chef servers updated in parallel while we test the new infrastructure

Story points: 3.0
Remove pgio from staging patroni servers

Exhibit 1:
* https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8026
* https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7932

Exhibit 2:
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6242#note_152545100

We're not using it, and it is causing problems.   It should be removed.

Story points: 1.0
Look into removing non-needed, and re-add needed, public IPs in gprd

After https://gitlab.com/gitlab-com/gl-infra/production/issues/1167, only the blackbox exporter and the console have public IPs in gprd.

Removing this IP from console would break kubernetes access, and removing it from the blackbox exporter broke monitoring for an unknown reason (possibly whitelisting).

Apparently these problems don't appear in gstg, but it's worth checking this out in case we're simply not noticing them.

If our own whitelisting, as opposed to customer whitelisting, is the reason we need these public IPs, then we should use static IPs and not exit via the Cloud NAT. This decouples Cloud NAT IP pool scaling from our infrastructure (isolates customer concnerns from our own).

One of the prometheus instances scrapes the CI runner managers and is federated with CI's prometheus. Firewall IP whitelisting in CI broke when we removed prometheus' public IPs, and was fixed by whitelisting the NAT IPs in CI. We should use a static IP for the prometheus instances to decouple this whitelisting from scaling the Cloud NAT IP pool.

Story points: 3.0
Chef not running on dev.gitlab.org

Plausibly broken related to the chef 12 vs chef 14 upgrade:

```
Sep 29 22:05:17 dev.gitlab.org chef-client[20093]: [2019-09-29T22:05:17+00:00] ERROR: cannot load such file -- highline
Sep 29 22:05:17 dev.gitlab.org chef-client[20093]: [2019-09-29T22:05:17+00:00] ERROR: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
cmiskell@dev:~$ ^C
cmiskell@dev:~$ dpkg -l |grep chef
ii  chef                                 12.22.5-1                                  amd64        The full stack of chef
```

@alejandro Any input into what the right way forward is?  I'd hate to break it more than it already is.

Story points: 1.0