giseldo/neodataset huggingface dataset:
Set up slack integration for access request workflow

Using the instructions from Gravitational, we need to install the Teleport Slack application: https://goteleport.com/teleport/docs/enterprise/workflow/ssh-approval-slack/

This app will relay access requests into the `#production-access` channel in Slack, and authorized users will be able to review and approve them.

Currently we are at the proof of concept stage, so this is still considered development, but there should be minimal changes to get this into production when we are ready.

Story points: 2.0
current Gitlab org Dockerhub plan expires Jan 31 2021

<!--
This template is for GitLab Team Members seeking support of SRE where there isn't an existing `request-*` template available.

Please fill out the details below.
-->

From Docker:

[Docker Team](https://email.docker.com/Ao6m0tFJ30IT3Po00K0x14L) plans start at $7 per user per month and provide access to advanced features and capabilities to help your team automate development workflows and increase productivity, including:
Unlimited private and public repositories
Unlimited authenticated image pulls
3 Parallel auto-builds
Role-based access controls
Unlimited teams
Unlimited Hub image vulnerability scans
And [much more](https://email.docker.com/Ao6m0tFJ30IT3Po00K0x14L)

You are receiving this email because your legacy organizational repository plan expires on your January 2021 billing cycle date. [Read the FAQ.](https://email.docker.com/y3oF4oPI0T0KLJy1tn00630)


cc @marin @erushton as I think this is legacy things we are still using dockerhub for.

I also see @ddavison, @tmaczukin and @twk3, @dzaporozhets as owners so cc in case they have further input on what we should do with this organization.


<!--
please do not edit the below
-->

Story points: 1.0
GitLab Hosted version of Codesandbox Sandpack Fork

### Overview

As part of https://gitlab.com/groups/gitlab-org/-/epics/3138#note_465738194 we needed to make some upstream contributions to the [Codesandbox Sandpack package `smooshpack`](https://www.npmjs.com/package/smooshpack). Unfortunately, it might take a while for these changes to land in the upstream package due to upstream availability.

After [receiving approval to fork this package](https://gitlab.com/gitlab-com/legal-and-compliance/-/issues/308), we've applied these changes to a [GitLab project](https://gitlab.com/pslaughter/gitlab-codesandbox-client) and [published a new npm package `gitlab-smooshpack`](https://www.npmjs.com/package/gitlab-smooshpack).

For us to [use this behind a feature flag](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50259#note_468978603), we need to have our GitLab forked package assets hosted in a separate bucket. Currently the Codesandbox package assets are hosted at `https://sandbox-prod.gitlab-static.net/`.

### Proposal

Similar to the [steps used for the original package](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/6709#note_259020854), here's how to get the specific assets we need to host:

```
# Download the package tarball
wget $(npm view gitlab-smooshpack@0.0.66-1 dist.tarball)

# Extract just the `sandpack` folder which contains the pre-built assets
tar -xzf gitlab-smooshpack-0.0.66-1.tgz package/sandpack

# The assets to host will be in `./package/sandpack`
```

Then drop all the files in `./package/sandpack` in a asset hosting bucket. Requirements:

* Suggested domain: `https://gl-sandbox-prod.gitlab-static.net/`
* SSL Enabled

### Links / References

- https://gitlab.com/groups/gitlab-org/-/epics/3138
- Legal issue for forking https://gitlab.com/gitlab-com/legal-and-compliance/-/issues/308

Story points: 3.0
Make sure pg_hba.conf is working for geo failover

During staging geo failover test, the last step of Promotion failed (geo:set_secondary_as_primary) attempting to authenticate with postgres:

```
2020-11-27_13:18:12.31820       Connection matched pg_hba.conf line 77: "host    all         all         0.0.0.0/0           md5"
2020-11-27_13:18:12.57502 2020-11-27 13:18:12 GMT [32557]: [1-1] FATAL:  password authentication failed for user "gitlab"
2020-11-27_13:18:12.57504 2020-11-27 13:18:12 GMT [32557]: [2-1] DETAIL:  Password does not match for user "gitlab".
```

It was temporarily fixed by adding

```
host    all         all         0.0.0.0/0           trust
```

above line 77 (which is mentioned in the error above) to `/var/opt/gitlab/postgresql/data/pg_hba.conf` and doing a `gitlab-ctl restart postgresql`, but this is getting overridden every time when doing a `gitlab-ctl reconfigure`.

We need to fix pg_hba.conf to work on both primary and secondary site for a failover or maybe need to change the DB password for the gitlab role.

Story points: 4.0
License app data export credentials configuration

(this issue description is based on [another similar issue's](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10670))

Request related to this issue: https://gitlab.com/gitlab-org/license-gitlab-com/-/issues/191

For automating export of the several tables for the `license-gitlab-com` project, we have a "pipeline Schedule" which needs several configuration parameters to be able to run properly.

- Configuration screen: https://gitlab.com/gitlab-org/license-gitlab-com/-/pipeline_schedules/85641/edit

## Credentials

| Variable | Value |
| ------ | ------ |
| GCLOUD_SERVICE_KEY | GCP [service account](https://console.cloud.google.com/iam-admin/serviceaccounts) auth JSON file name |
| GOOGLE_PROJECT_ID |  GCP project id ( API [sqladmin.googleapis.com] not enabled on project) ) |
| GOOGLE_COMPUTE_ZONE | The region and zone for the GCP compute |
| BUCKET | The GCS bucket where we shall upload the SQL export |
| INSTANCE | Cloud SQL instance name |
| DATABASE | Cloud SQL database name |

Story points: 1.0
Migrate Artifacts from S3 to GCS

We need to migrate artifact object storage from S3 to GCS. This has the added benefit of allowing `direct_upload` of artifacts. Much of the work of this has already been done. We sync the GCS bucket with S3 daily so everything is kept up do date. The process for completing the migration should be as follows:

- [ ] Disable `direct_upload` of artifacts
- [ ] Run a final sync between S3 and GCS of the artifacts bucket
- [ ] Change storage location in `gitlab.rb` to point to GCS
- [ ] Verify artifacts still work
- [ ] Enable `direct_upload` of artifacts once more
- [ ] migrate the artifacts that landed on disk to object storage

cc/ @ayufan @jarv @dosuken123

Story points: 3.0
Create a canary in gprd/gstg

We don't have a canary environment in gprd yet and we probably should. I think it would be nice to get this done pre failover if possible but for now it will be put in the post-failover milestone.

Here is a summary of the current proposal:

- Do not use a new chef environment, use the existing gprd/gstg environments
- Have the option to create a gstg cny so we can test configuration changes on gstg before gprd
- The cny names will be `{web,api,git}-cny-##-sv-{gprd,gstg}`
- First iteration will just have a single web node, this means we will have the following two new vms
  - web-cny-01-sv-gstg.c.gitlab-staging-1.internal
  - web-cny-01-sv-gprd.c.gitlab-production.internal
- We should run migrations for the canary deploy
- To identify them in prometheus we will use a new tag stage=canary
- Will need to make some takeoff adjustments since canary is no longer its own environment
- Once we have the canary accepting traffic with a cookie set in the browser, we will add weights to the haproxy configuration so that a small percentage of production traffic will go to the canary

For the work here is what will need to be done:

- [x] Create new shared terraform config for canary
- [x] Create new chef configuration for the canary roles (gstg, then gprd)
- [x] Modify takeoff so that we can deploy to canary
- [x] Create canary specific dashboards
- [ ] 
Create canary specific alerts
 Will track this separately, when we start moving some prod traffic to the VM.
- [ ] 
Update the gitlab-haproxy config so that some production traffic is directed to the canary (this doesn't need to be in the first iteration)
 We will track this separately.





Story points: 4.0
Drop InfluxDB entirely

This issue is to capture the effort to remove InfluxDB from our infrastructure once our Prometheus metrics are good enough, and obviously post GCP-migration.

More conversation is here: https://gitlab.com/gitlab-com/migration/issues/145#note_55954264

Check the tickbox if you're satisfied....

- [ ] **CI Runners**: https://dashboards.gitlab.net/dashboards/f/bpcbFeIiz/ci-runners-service cc @tmaczukin
- [ ] **GCP Migration Project** https://dashboards.gitlab.net/dashboards/f/vtC8ceSmk/gcp-migration-project cc @andrewn
- [ ] **Gitaly** https://dashboards.gitlab.net/dashboards/f/SRXyrrSmk/gitaly-service cc @tommy.morgan
- [ ] **GitLab Rails** https://dashboards.gitlab.net/dashboards/f/KuFJt6Iiz/gitlab-rails-service cc @smcgivern
- [ ] **HA Proxy** https://dashboards.gitlab.net/dashboards/f/N9YSt6Siz/haproxy-service cc @northrup
- [ ] **Operations** https://dashboards.gitlab.net/dashboards/f/KN7OgCSiz/operations cc @dawsmith
- [x] **Pages** https://dashboards.gitlab.net/dashboards/f/v2ZhpeSik/pages-service cc @nick.thomas
- [x] **Postgres** https://dashboards.gitlab.net/dashboards/f/u3xKRjIiz/postgres-service cc @yorickpeterse
- [ ] **Prometheus** https://dashboards.gitlab.net/dashboards/f/5dsvpeImz/prometheus-service cc @bjk-gitlab
- [ ] **Redis** https://dashboards.gitlab.net/dashboards/f/D5R0peIik/redis-service cc @jarv
- [x] **Workhorse** https://dashboards.gitlab.net/dashboards/f/_OKcteIiz/workhorse-service cc @jacobvosmaer-gitlab

FYI, several other folders exist:

- **Broken** https://dashboards.gitlab.net/dashboards/f/cYFN9rSmz/broken Dashboards that rely on Influxdb and are not currently working

- **Once-off** https://dashboards.gitlab.net/dashboards/f/kGgx9rIiz/once-off Once off dashboards

- ~~**Archives** https://dashboards.gitlab.net/dashboards/f/phfSe3Iiz/archived @northrup and @toon's manually migrated dashboards (first attempt)~~ (cleaned and deleted)


/cc @sytses @jarv @andrewn @jtevnan @northrup @ilyaf @ahanselka

Story points: 4.0
[META] Structured (JSON) logging for PostgreSQL

Structured logging would make it easier to ingest and search through logs. In particular this could be useful for searching for slow queries that may spread across multiple lines. We should investigate the options here and if deemed possible/useful enough implement it.

There are several steps to investigating and deploying this change:

* [x] Build the jsonlog module and manually install it on staging
* [x] Decide on the destination path for the logs
* [x] Evaluate the log output and fix any deficencies
  * [x] https://github.com/michaelpq/pg_plugins/pull/18
  * [x] https://github.com/michaelpq/pg_plugins/pull/19
  * [x] https://github.com/michaelpq/pg_plugins/pull/20
  * [x] https://github.com/michaelpq/pg_plugins/pull/21
* [x] Configure ELK stack to ingest these logs
  * [x] https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/merge_requests/5#note_61734780
  * [ ] Ensure these logs are rotated and cleaned up after some retention period
  * View 
* [ ] Integrate jsonlog into omnibus
  * https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2332#note_62951392
* [ ] Wait for omnibus with jsonlog to be deployed to production
* [ ] 


Unanswered questions:
* [ ] Can we still use mtail or is there an alternate way to get metrics from logs?

Story points: 4.0
[meta] New Geo Testbed

## Updated plan with smaller issues

Azure - geo1.gitlab.com:

- [x] ~1d Create terraform configuration for Azure - https://gitlab.com/gitlab-com/infrastructure/issues/3090
- [x] ~2d Create new chef environment for Geo testbed - https://gitlab.com/gitlab-com/infrastructure/issues/3091
- [x] ~2d Setup NFS storage for the geo testbed in Azure (copy of nfs shards) - https://gitlab.com/gitlab-com/infrastructure/issues/3092
- [x] ~1d Setup Uploads/Attachments for the geo testbed in Azure - https://gitlab.com/gitlab-com/infrastructure/issues/3093
- [x] ~1d Setup Database that is pseudo-anonimized and only contains projects for the nfs shards  -https://gitlab.com/gitlab-com/infrastructure/issues/3094
- [x] ~1d Setup monitoring and alarming for the Azure testbed - https://gitlab.com/gitlab-com/infrastructure/issues/3095
- [x] ~1d Setup automatic deploys for the Azure testbed - https://gitlab.com/gitlab-com/infrastructure/issues/3096

Azure - geo2.gitlab.com:

- [x] ~1d Create terraform configuration for Azure - #3203 
- [x] ~2d Create new chef environment for Geo testbed - #3204
- [x] ~2d Setup NFS storage - #3205
- [x] ~1d Setup Uploads/Attachments - #3206
- [x] ~1d Setup Database with disk config as geo1 - #3207
- [x] ~1d Setup monitoring and alarming - #3208
- [x] ~1d Setup automatic deploys - #3209


## Original issue

Per discussions a few days ago, we'd like to have a new Geo testbed (or upgrade the existing one) so that it has GitLab running in HA mode on both the primary and secondary, with at least two web workers on each side, and load balancer in front, to make sure that none of the Geo code / functionality bugs out in an HA setup. 


Story points: 5.0
Set up alert on HTTP Queue Timing

Carry over from https://gitlab.com/gitlab-com/infrastructure/issues/2379

While we anticipate better ways of measuring web request response timing with more integration of Prometheus, we should not wait on that to set up an alert on HTTP Queue timings. Once those timings go above an SLO that I am about to propose in this issue, the runbook that ties to the alert should describe how to add more unicorn hosts, as a first step. Obviously there can be other reasons for high HTTP Queue Timings, but this is a start.

- Proposed SLO: HTTP Queue Timing p99 < 15 ms.
- Proposed alert: Alert when p99 has been > 15 ms for more than 30 minutes.

https://performance.gitlab.net/dashboard/db/transaction-overview?panelId=13&fullscreen&orgId=1

Story points: 5.0
Sync `jws_private_key` between all app servers

Zendesk: https://gitlab.zendesk.com/agent/tickets/79645

The person that contributed the OpenID Connect feature reported that GitLab.com is returning a different key for each request, presumably because the app servers don't have the private key synced. We should sync `jws_private_key` in `gitlab-secrets.json` across all nodes. 

See https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/1222/diffs for where this was implemented.

Story points: 2.0
Create a Terraform gym

As a follow-up of #1715 it'd be great if we can create a Terraform gym to fiddle with the tool in an isolated environment. This to allow team members to learn and experiment in complete freedom on the same platform as production without the risk of hurting GitLab.com.

To do this we should create a separate Azure account (https://gitlab.com/gitlab-com/infrastructure/issues/1906) and decide if we want to limit the exercises to pure infrastructure or if we can spend some time and have a Chef server set up for this. The latter would allow full coverage of the way we provision servers at the moment. Discussion about this from [this comment](https://gitlab.com/gitlab-com/infrastructure/issues/1715#note_31536847). Let's continue it here.

/cc @pcarranza @ernstvn @northrup @jtevnan @stanhu

Story points: 5.0
Automate the staging environment creation

Following a call with @ernstvn and @andrewn we agreed that we need a way to create staging environments for testing new features. This is meant to be an intermediate step towards #1504 which remains the ultimate long term goal.

The way we can achieve this in the short to mid term is to leverage on Terraform for the creation of the nodes. This is already part of an existing effort to better align staging to the production environment and the configuration [is already coded](https://gitlab.com/gitlab-com/gitlab-com-infrastructure/tree/master/staging).

To make it scalable to a number of different staging environments we will need to define a schema for the chef roles, like `staging-<name>-*`.

@andrewn was also asking about how we would deal with the monitoring, since it's a fundamental component to gather data about the changes. Would we use a global influxdb/prometheus server? Should we use one for all staging environments? Should we create one for each?

Last but not least, the database. We'll probably need to use a copy of the staging data set. If we use the same staging database then the schema will get stale real quick. Also, this could be a good opportunity to test migrations in a controlled way. @northrup has a plan for that.

/cc @gl-infra  @bjk-gitlab

Story points: 5.0
Custom robots recipe applies multiple lines

The [gitlab-server::custom-robots](https://gitlab.com/gitlab-cookbooks/gitlab-server/merge_requests/46/diffs) recipe seems to apply the `Disallow: /snippets/*` line multiple times to `robots.txt` on production. 

**Symptoms**

```
# tail /opt/gitlab/embedded/service/gitlab-rails/public/robots.txt
Disallow: /snippets/*
Disallow: /snippets/*
Disallow: /snippets/*
Disallow: /snippets/*
Disallow: /snippets/*
Disallow: /snippets/*
Disallow: /snippets/*
````

__________________

The issue is probably due to the regex 

```
robots.insert_line_if_no_match("/Disallow: \/snippets\/\*/", "Disallow: /snippets/*")
```

Story points: 1.0
Redirects on blue-moon and status.gitlab.com

We need to create a new node which will be chef controlled and which will hold all the redirects to the domain.
Old domains still point to the blue-moon IP address and this redirects to the current website.
Blue-moon also holds status site.

To discontinue blue-moon, we need to:

- [ ] Move status to a separate node
- [ ] Provision a new node to host all the redirects
- [ ] update the IP for all domains listed in `/etc/nginx/sites-available/www.gitlab.com`
- [ ] Make sure that the redirects in `blog_rewrite_rules.rules` and `org_to_com_rewrite_rules.rules` are still working

cc @jnijhof @pcarranza 

Story points: 5.0
Add more runner capacity for ops.gitlab.net

We currently have three runners
* one dedicated for release
* one dedicated for chatops
* one for everything else.


The one for everything else I think would be nice if we could switch to kubernetes.
We should probably create a dedicated runner for traffic generation
The runner cookbook isn't designed for how we are using it now with multiple runners so we may want to make some minor updates to it. Ideally we can scale each runner type out horizontally with different tokens.

Story points: 2.0
Rake task rake update_vault_admins does not appear to work

In the setup of new SRE's we need to execute the rake task `rake update_vault_admins` to update the vaults such that the user has their new access to our vaults.  This does not appear to be working...

The vault is not updated with the new users.  A very quick look doesn't really indicate to me why this task would not work.

Story points: 1.0
META: Move backup CICD jobs needed for operations to ops.gitlab.net and switch to pushgateway for monitoring

We will probably want to split this out into the following smaller tasks:

- [x] Move https://gitlab.com/gitlab-restore to `gitlab-com/gl-infra/` first. I'm not sure why they are under this `gitlab-restore` group but we should keep everything under `gitlab-com` so it was a mistake for sure. 
- [x] Mirror the repositories in  `gitlab-restore` over to ops.gitlab.net with a push rule on the projects under `esc-tools`.
- [x] Move the CICD configuration from gitlab.com to ops.gitlab.net. This means we will use gitlab.com as the source for code but CICD will run on ops.
- [ ] Replace deadmansnitch with pushgateway for notifications and alerts -> https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5167

Story points: 2.0
Update the status page from chatops

It was noticed on https://gitlab.com/gitlab-com/gl-infra/production/issues/564 that we did not update our status page. One of the reasons for this is that it is a job currently done manually by the MOC and is sometimes overlooked.

This needs to be easier, all of our status updates that are posted via twitter should go through the status page first as it is the most likely place for the wider community to check for what is going on.

Story points: 5.0
Access To Sidekiq Rails Logs For Staging

I am trying to read `Rails.logger.error` logs from staging sidekiq worker jobs. I have an example log in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/services/clusters/applications/install_service.rb#L16 which I'm sure is being triggered on staging but no such log is appearing in Kibana. I've tried searching `"Kubernetes error"` in `pubsub-sidekiq-inf-gstg*` but nothing shows up for the last 7 days and I've triggered it many times.

If we aren't forwarding these logs to kibana then I need access to some other system that has these logs as they are necessary for me to troubleshoot issues in staging.

I have yet to be able to determine if these logs are available in kibana for production either as we haven't deployed that code to production yet but assuming they aren't available in production kibana then I'll also need access to the production logs so I can read them there too.

Story points: 2.0
Rollout Prometheus 2.5

Prometheus [2.5.0 is out](https://github.com/prometheus/prometheus/releases/tag/v2.5.0). Now includes a `--query.max-samples` flag to limit how much memory a query can use.

* [x] Decide on a max-samples limit.
* [x] Rollout to staging.
* [x] Rollout to ops.
* [x] Rollout as default.

Story points: 2.0
Log DDL statements with txid

In order to recover from a `DROP TABLE` or otherwise DDL-related incident without data-loss, we employ PITR to replay until right before the incident happened. Knowing the txid of the incidental transaction enables us to precisely replay until the last transaction before the incident. Without knowing the txid, one would have to replay until a certain point in time which is loss-inducing (if too early) or rendering the restore unusable (if too late).

So the proposal here is to:

* log DDL statements (`log_statements = 'ddl'`)
* make sure the log prefix contains the transaction id

Story points: 1.0
Change label title

Because we strive not to use violent analogies in our culture and language, we need to update the label "outage:postmortem" as per https://gitlab.com/gitlab-com/www-gitlab-com/issues/2821. I'd recommend we change it to "**outage:rootcause**" or "**outage:analysis**" for the sake of brevity

I found the label [here](https://gitlab.com/gitlab-com/gl-infra/infrastructure/labels?utf8=%E2%9C%93&subscribed=&search=postmortem), but I don't believe I have the permissions necessary to edit it.

Story points: 1.0
Ingest structured audit logs into ELK

With https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22471 we now have structured audit logs coming from rails :tada: (thanks @stanhu!)

These logs are even flowing into stackdriver.

We should also ingest them in ELK.

Once we have them, we'll be able to use them for a whole lot of abuse detection activities in the way we do for Gitaly at present.

cc @jarv 

cc @dawsmith and @Finotto for scheduling

Story points: 1.0
Add alert for increase in active sessions

as a ~"corrective action" for https://gitlab.com/gitlab-com/gl-infra/production/issues/553 we should add an alert for a large increase in active sessions.


https://dashboards.gitlab.net/d/wyBYDjnmk/gitlab-workhorse?panelId=8&fullscreen&orgId=1&from=1541446106294&to=1541526372156

![Screen_Shot_2018-11-06_at_6.46.53_PM](/uploads/7e141f35d147f77bbbcd780aa7b7025e/Screen_Shot_2018-11-06_at_6.46.53_PM.png)

Story points: 2.0