giseldo/neodataset huggingface dataset:
Create a new wildcard certificate and domain for `*.gitlab-ce-review.app`

From the meeting we had today on addressing cleanup and scalability, the team would like to create a new cluster for CE in the review apps project.
* Meeting notes: https://docs.google.com/document/d/1oLc4s02U_bNxx2UCO-dHw5C3swhNI8QCYpX6taNnSi8/edit#heading=h.nzenrwnechk2

Currently, we are using the one created in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4735 for both CE and EE.

This has proven to be a challenge with debugging and implementing cleanup/scale. 

We would like to use 2 clusters so we have some degree of separation. The load from CE is a lot more than EE. The benefit of having 2 clusters:
* Separation of stability, if the clean up of CE is unstable EE is not affected
* Ease of debugging during our first phases of implementation.
* Possibility of having a more aggressive clean up for CE (at least in the short term)

/cc @marin @ibaum @rymai @ddavison 

@dawsmith if you can help us assign this to someone that would be much appreciated :pray: I set the weight to 2 similar to the request we had before (linked above)

Story points: 1.0
Redis cache failover can lead to data corruption

Currently `SAVE` is disabled on the Redis cache instance. 

This change has been in place since January 3rd 2018: https://dev.gitlab.org/cookbooks/chef-repo/commit/a60bdeded9fbd1cb295db501b4f4f18e27dabb54

When sentinel detects a reboot, it may decide to wait for the original master to come back online rather than failing over to a secondary. We experienced this during 
 Saturday's planned failover. It's mentioned in several Redis issues (see below) but the reasoning for this behaviour is not yet clear to me.

When coupling this with our current setup, the following scenario may occur:

1. Redis has a snapshot file in `dir` with an unspecified date.
1. We reboot a Redis master
1. Since we have `save ""`, Redis will shutdown without saving the snapshot (by design)
1. Redis will restart. On start it will begin loading the RDB snapshot file. This file might be weeks old.
1. Load complete, the slaves resync off the master, overwriting their "current" cache with an old, invalidated cache
1. The application requests a cache item and is returned previously invalidated data. This data is then written to a persistent store (eg postgres) leading to further corruption.

See
* https://github.com/antirez/redis/issues/1297
* https://github.com/antirez/redis/issues/1281

cc @jarv @dawsmith @Finotto

Story points: 3.0
Tune threshold for sidekiq exception alerts

We recently created a new alert for sidekiq exception count in #4651, but quickly noticed that we were seeing radically different volumes (~5x to ~15x higher) from one controller in particular, `RepositoryUpdateMirrorWorker`. After looking at https://sentry.gitlab.net/gitlab/gitlabcom/issues/217714/, it looks like the majority are for legitimate, albeit misconfigured jobs.

For an easy fix to make this alert more useful and cut down on the noise, we should implement a separate threshold for the `RepositoryUpdateMirrorWorker`. Based on https://dashboards.gitlab.net/d/9GOIu9Siz/sidekiq-stats?panelId=66&fullscreen&orgId=1&from=now-30d&to=now, setting it to `10000` for `RepositoryUpdateMirrorWorker` and `2000` for the others should be more appropriate

Story points: 3.0
Ship new .log files to Elasticsearch

In 11.3, we added new JSON files:

* `/var/log/gitlab/gitlab-rails/integrations_json.log` (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21316)
* `/var/log/gitlab/gitlab-rails/importer.log`

We should ship these to Elasticsearch/Stackdriver etc.

Story points: 1.0
Network issues caused connectivity problems for about 2 minutes

# Summary

We had a network blip today that caused some issues connecting to various services.


# Timeline of events

Pagerduty alerts
* 7:17am gitlab.com issue is down
* 7:18am gitlab.com issue alert cleared
* 7:18am gitlab.com pages is down
* 7:18am gitlab.com new repo is down
* 7:19am gitlab.com new repo alert cleared
* 7:19am gitlab.com pages alert cleared

During this time, we also had trouble logging in to servers.  It cleared up on its own.


## Monitoring


This appears to be due to a network glitch

![image](/uploads/d819a4ac6c716832086acf7297b2edce/image.png)
Separate gitlab-pages from the rest of the web worker fleet

Currently GitLab pages runs alongside unicorn on all web-workers on `web-*` nodes.

The traffic profile for GitLab Pages, and it's growth, is not related to the profile of the rest of the application, but by having this service running alongside the web, we have coupled the scaling of the Rails web service to the Pages service.

This has many disadvantages:

* We have 14 instances of the pages service running alongside 14 web workers. This is vastly more than we need. 
* Each pages daemon loads the index into memory and is competing with the unicorn fleet on memory consumption, even when traffic volumes (per instance) are low.
* Pages deployment and restart is notoriously slow. Having unnecessary additional workers exasperates this problem.
* Pages requires additional NFS mounts that the web workers would otherwise not require, so splitting this would allow us to drop those NFS mounts.


cc @glopezfernandez @dawsmith @Finotto @jarv

Story points: 2.0
PullMirrorsOverdueQueueTooLarge from staging triggering pager duty

The PullMirrorsOverdueQueueTooLarge from gstg was triggering pagerduty. 

This MR removed the filtering so that alert manager could decide whether to page.  
https://gitlab.com/gitlab-com/runbooks/merge_requests/733

Since alert manager is making the wrong decision, I've added this MR to remediate the immediate problem, but we'll need a better solution.  https://gitlab.com/gitlab-com/runbooks/merge_requests/759

The pagerduty alert looked like:
```
Labels:
 - alertname = PullMirrorsOverdueQueueTooLarge
 - channel = backend
 - monitor = gstg-app
 - pager = pagerduty
 - provider = gcp
 - region = us-east
 - replica = 01
 - severity = critical
Annotations:
 - description = On average, there have been over 5000 overdue pull mirror jobs for the last 10 minutes. Check https://dashboards.gitlab.net/d/_MKRXrSmk/pull-mirrors.
 - runbook = troubleshooting/large-pull-mirror-queue.md
 - title = Large number of overdue pull mirror jobs: 13343
Source: https://prometheus-app.gstg.gitlab.net/graph?g0.expr=quantile%280.5%2C+gitlab_database_rows%7Bquery_name%3D%22mirrors_ready_to_sync%22%7D%29+%3E+5000&g0.tab=1
Labels:
 - alertname = PullMirrorsOverdueQueueTooLarge
 - channel = backend
 - monitor = gstg-app
 - pager = pagerduty
 - provider = gcp
 - region = us-east
 - replica = 02
 - severity = critical
Annotations:
 - description = On average, there have been over 5000 overdue pull mirror jobs for the last 10 minutes. Check https://dashboards.gitlab.net/d/_MKRXrSmk/pull-mirrors.
 - runbook = troubleshooting/large-pull-mirror-queue.md
 - title = Large number of overdue pull mirror jobs: 13342
Source: https://prometheus-app.gstg.gitlab.net/graph?g0.expr=quantile%280.5%2C+gitlab_database_rows%7Bquery_name%3D%22mirrors_ready_to_sync%22%7D%29+%3E+5000&g0.tab=1
```

Story points: 1.0
DashboardsGitlabComDown Alert

dashboards-com-01-inf-ops.c.gitlab-ops.internal had the grafana-server process crash.  Running chef manually did not bring it back up.  Rebooted the server.  The process came back up with the server.  The alert cleared a few minutes later.

I wasn't sure how to determine why the process crashed. @jarv, can you help figure out why this happened?

Story points: 1.0
Add SSL Version to HA Proxy Logging

The SSL Version of the handshake is good information to track, especially as we look to continue to make improvements to our SSL handshake by removing weak and deprecated methods.

### Things to Do
- [x] Add `%sslv` in https://gitlab.com/gitlab-cookbooks/gitlab-haproxy templates after `%ft`
- [x] Update FluentD config for added field in HA Proxy Logs
- [x] Ensure Stack Driver ingestion without issue.

Story points: 5.0
Requesting specific permissions on GCP gitlab-demos project

For my partner demonstration work I need to be able to test install GitLab from the Google MarketPlace on GCP. I'm requesting that my account (dgordon@gitlab.com) be given proper privileges so that I can do this.

I'm doing this to a cluster I have created in the gitlab-demos project. When I try to install from MarketPlace it gives me the error that I need "Kubernetes Engine Admin" permissions.

![image](/uploads/101584b0c38a8aafb06041490e2e6713/image.png)

  Thanks

Story points: 1.0
Redirect a couple of whitepapers

We're in the midst of moving some content behind email gates and I'd like to redirect the old URLs to the new landing pages. Here are the details:

| old URL | redirect to URL |
| -------- | -------- |
| /pdfs/resources/gitlab-moving-to-git-whitepaper.pdf  |  /resources/whitepaper-moving-to-git/  |
| /pdfs/resources/gitlab-scaled-ci-cd-whitepaper.pdf   | /resources/whitepaper-scaled-ci-cd/   |

Thanks!

Story points: 1.0
Plan for making ops.gitlab.net the source of truth for operations repos for chef and terraform automation

Given that we are automating many of our infrastructure repos with CICD on the ops.gitlab.net instance I think it is probably time to consider switching to ops.gitlab.net as the source of truth for:

* `https://gitlab.com/groups/gitlab-cookbooks/*`
* https://dev.gitlab.org/cookbooks/chef-repo
* https://gitlab.com/gitlab-com/gitlab-com-infrastructure

### Reasoning

* The deployment that we submit MRs to should be the same deployment that runs the CICD jobs, this is important for approving and reviewing.
* We want to encourage contributions to these repositories and ensure that there is very little friction for gitlab.com team members to contribute. 
* We currently have https://ops.gitlab.net setup so that anyone with a gitlab.com email address can login and view protected repositories.  
* All repositories that were on gitlab.com will be mirrored on gitlab.com
* The downside of this is that this plan does not have a way for the wider community to contribute, there won't be an easy way to enable this without enabling the wider community to create accounts on ops.gitlab.net which may be an option we later consider.

This is a proposed high level plan on how to do this in a sane way:

# Plan
## chef-repo
- [x] Prevent pushes to all branches in settings
- [x] Export / Import the project into ops.gitlab.net
- [x] Notify members to update their remotes
- [x] Setup the repository to push to gitlab.com/gitlab-cookbooks/chef-repo (private project)
- [x] Update description so that it is clear where the source is

## gitlab-com-infrastructure
- [x] Prevent pushes to all branches in settings
- [x] Export / Import the project into ops.gitlab.net
- [x] Notify members to updates their remotes
- [x] Setup the repository to push to gitlab.com/gitlab-com/gitlab-com-infrastructure (public project)
- [x] Update description so that it is clear where the source is

## gitlab-cookbooks/*
- [ ] Change group membership for gitlab-cookbooks so only ops-gitlab-net can push
- [ ] Update the repository settings on ops.gitlab.net for all gitlab-cookbooks repositories so that they are configured to push to gitlab.com using the ops-gitlab-net user
- [ ] Notify members to updates their remotes\
- [ ] Update description so that it is clear where the source is

cc @gitlab\-com/gl\-infra @andrewn @dawsmith @gerir @marin

Story points: 3.0
[Design Document] First iteration of kubernetes migration

It was brought up during our last on-call handover that we should be careful about the opportunity cost of spending time on efforts (such as internal network hardening) that may not be relevant as we start migrating services to kubernetes.

Personally, I find this cost difficult to reason about given we do not have yet a roadmap or plan for a first iteration of kubernetes service migration for gitlab.com. This issue is to track just that, a design document for what we can accomplish as a first iterations for kubernetes and a horizon for completing the work against other priorities.

Currently I believe the first likely candidate for k8 will be the registry service given that we have already set it up once and that it is a standalone service. The next service might be sidekiq. I propose we document a transition plan for registry and sidekiq unless there are other services that are a better fit.

I will put this into the next milestone for consideration, unless we decide it is not something we want to start thinking about.

cc @andrewn @dawsmith @gerir @Finotto @bjk\-gitlab @jurbanc

Story points: 2.0
the restore pipeline for disk snapshots is broken

Taking snapshots is working but the restore pipeline is broken:

https://gitlab.com/gitlab-restore/gitlab-production-snapshots/-/jobs/101608916

- [x] Fix the pipeline stage so that if it fails it fails the pipeline step
- [x] Use deadman snitch so that this pipeline is better monitored
- [x] Fix the restore pipeline


I don't think this is extremely urgent as the snapshot pipeline is working, just the restore test is broken. I will put it in the next milestone since this one is already at capacity.

cc @Finotto @dawsmith

Story points: 2.0
New bastion host in AWS for Registry analysis

Spun out of https://gitlab.com/gitlab-org/gitlab-ce/issues/51702#note_104213310

> @andrewn Can you spin-up the bastion host (likely best on AWS) with 64GB of RAM and S3 read-only credentials that give GET and LIST permission?
> Once I start doing this "manual scan" I will start writing steps needed to do that.

cc @ayufan 

cc @dawsmith @Finotto for scheduling

Story points: 2.0
Sidekiq is not being restarted after chef deploy for customers.gitlab.com

REF: https://gitlab.com/gitlab-org/gitlab-ee/issues/7518

I was trying to fix a bug related to one of our background jobs but realized that after running `sudo chef-deploy` the workers were still using an old version of the app, so I was required to do a manual restart in order to see the fix working.

Can we fix it through the [cookbook](https://gitlab.com/gitlab-cookbooks/cookbook-customers-gitlab-com)?

/cc @jarv @northrup

Story points: 3.0
Setup web, api and git canary to receive a small percentage of web traffic

This task is part of the canary project, outlined in the design doc https://docs.google.com/document/d/15jzLb5O4ASPYInxw1CFI-ngLHv2K8w01AuFOkejfcrk/edit#


- [x] Create the infrastructure for web/api/git
- [x] Create utilitiesthat will adjust the weight of these servers so they can receive a small portion of gitlab traffic

Story points: 4.0
Configuration updates to introduce optional backend weights to the haproxy configuration

Described in the design doc - https://docs.google.com/document/d/15jzLb5O4ASPYInxw1CFI-ngLHv2K8w01AuFOkejfcrk/edit#

We will want to have an option for all backends to introduce weights and also optionally include the canary hosts with a backend weight of zero.

Story points: 2.0
[Design Document] GEO for DR

Please fill with your ideas the following design doc: https://docs.google.com/document/d/1_fuskON5fbZgxoBktSp-vFvHUU8IauX-q_YWhnT4J5k/edit?usp=sharing

AC - switch to doing an MR in the handbook rather than the google doc.

Story points: 1.0
Add PagerDuty webhook to open a GitLab incident issue for all PagerDuty incidents

Currently we have a fairly clumsy process where we get a page and either open an incident manually in the production tracker or in the case where GitLab is down we use the slack bot to initiate it.

I don't think we need to change the latter but for the former I think we can automate this in a way that we don't have to open up issues manually. 

I think the best option for us is to probably use a google cloud function webhook to handle the api request to open an issue on an incident. What we should include is a way to avoid spamming the infrastructure tracker when incidents fire again for the same event. For this we can probably add an api query to see if an issue is already open before opening a new one.

cc @andrewn @jurbanc 

Tasks (updated by @peterdam)
* [x] Understand what information GitLab production issues contain
* [X] Review recent PagerDuty incidents to understand what information is captured [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_119676468)
* [x] Understand the conditions for which a GitLab production issue must be created i.e. requirements. [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_119591691)
* [X] Figure out which GCP Project to use - **Preference is gitlab-ops** [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_120432576)
* [x] Figure out what is the team's preferred FaaS provider (GCP or AWS) - **Preference is GCF** [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_119628319)
* [x] Understand [alertmanager-slack-bridge](https://gitlab.com/gitlab-com/gl-infra/slackline)
* [x] Understand [security pager](https://gitlab.com/gitlab-com/security-tools/security-pager) - **Not relevant for this issue** [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_120454499)
* [ ] Determine service accounts to be used when connecting to GitLab, maybe we have a generic service account
* [x] Read up on GitLab REST API
* [X] Create flow diagram as proposal [Notes](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5121#note_120636590)
* [x] Enable GCF API for `gitlab-ops` Google Project
* [ ] Write function/code in Python3.7
* [ ] Create PagerDuty Webhook which will connect to `https://us-east1-gitlab-ops.cloudfunctions.net/<function_name>`
* [ ] Deploy
* [ ] Look into Zapier - PagerDuty/GitLab integration - Seems like the easy option
* [ ] ... more to come ...



Story points: 2.0
Introduce suspension of changes and releases during important events

During the livestream on 2018-09-20 we decided to suspend a [network configuration change](https://gitlab.com/gitlab-com/gl-infra/production/issues/478) and I think it might be worth discussing a way to formalize this process.

What we need:

1. something to call it
2. something that is accessible for the team and company to view
3. something we can query from CICD to stop our automated changes from rolling out


For (1) we could call it Production SOC (suspension of changes) though it is not a standard term. We should avoid other popular terms like "blackout windows" or "blackdays". I did some googling around and couldn't find anything that is standard and would love to hear suggestions.

For (2) I would like to signal boost this but putting it on the shared meeting calendar, if that is a bit too much we could create a new calendar for it. 

For (3) would a google calendar work or should we consider other options?
the last internal company pipeline tool I used had a feature where you could force pipelines to pause during the certain time intervals, @ayufan is there anything like this on the roadmap for cicd?

## Name suggestions (feel free to vote or add more if you have them)

* SOC (suspension of changes) :ghost: 
* Change suspension :palm\_tree: 
* office hours :roller\_coaster: 
* ICL (Infrastructure Change Lock) :8ball: 
* Change freeze :snowman:

cc @gitlab\-com/gl\-infra @glopezfernandez @andrewn

Story points: 3.0
CI/CD for terraform (require apply stages)

To close out the final two implementation points in #4872: after #5114 has been successfully implemented and validated, we need to update the pipeline in gitlab-com/gitlab-com-infrastructure> so that the apply stages have the setting `allow_failure: false` to ensure that they are [blocking manual actions](https://docs.gitlab.com/ee/ci/yaml/#when-manual), and are required before merging to master.

Story points: 1.0
CI/CD for terraform (tf apply)

Per the high-level plan in #4872, we need to update the CI/CD pipeline for gitlab-com/gitlab-com-infrastructure> to [manually run](https://docs.gitlab.com/ee/ci/yaml/#when-manual) `tf apply` for each environment, starting with `gstg`

Also see [this section](https://www.terraform.io/guides/running-terraform-in-automation.html#multi-environment-deployment) of the terraform documentation for more details about automating deployment to multiple environments. Since we are not (currently) using terraform workspaces, the MVC first pass should probably just have optional manual steps for each production environment after `gstg` (i.e. `ops` and `gprd`) using the build artifacts from #5113. Eventually we may need to consider re-organizing repositories along the lines of [this comment](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4872#note_100185346).

Story points: 1.0
CI/CD for terraform (tf plan)

Per the high-level plan in #4872, we need to update the CI/CD pipeline for gitlab-com/gitlab-com-infrastructure> to run `tf plan` for each environment during a "build" stage, with a build artifact of the plan file and `.terraform` directory for each environment that can be re-used for automated `tf apply` stages later.

See [this section](https://www.terraform.io/guides/running-terraform-in-automation.html#plan-and-apply-on-different-machines) of the terraform documentation for more details.

Story points: 1.0
301 redirect for a page on the about website

Please infra team, can you set up a 301 from https://about.gitlab.com/blog/categories/release/ to https://about.gitlab.com/blog/categories/releases/?

The former is a 404 due to renaming a blog category from `release` to `releases`. cc/ @rebecca FYI for future changes and so that you can check if we need more redirects from old categories.

It caused some backslash on HN as we linked that page from the docs. See https://gitlab.slack.com/archives/C16HYA2P5/p1537391564000100 for reference.

Story points: 2.0
Apply receive_max_input_size setting on GitLab.com

@rdavila's change to limit the max push size has been merged: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20758 (thanks @rdavila :+1:!)

See https://gitlab.com/gitlab-org/gitlab-ce/issues/26044 for further details.

This was a production request, and an important control for improving the stability of GitLab.com.

See should apply this control to GitLab.com. What is a reasonable maximum size?

cc @jarv @dawsmith @Finotto

Story points: 1.0
Switch GitLab.com Early Adopters to Free on October 1st, 2018

We extended the Early Adopter program to Oct 1st (see https://gitlab.com/gitlab-com/marketing/general/issues/3050) and emailed impacted users around Sept 19th.

On October 1st, we should switch all Early Adopter plans to Free plans on GitLab.com.

Story points: 2.0
2018 - standups and planning updates

Proposing making a slack channel for standups and planning to have a record / place for team communication on these topics.

~~Channel name:  infra-standup~~

Based on comments- hold off on making the channel and just use # sre-lounge

### Standups: 

* [x] todo: setup the bot to remind 

they should cover: 
What are you working on?
What are you blocked on?
Opportunities to pair up?
Things I worked on that can be handed off?

### Milestone Planning:
Thinking about how best to make milestone and issue level planning async.
The current thought is to use the standup channel for posts to request feedback on issues that need to be planned in the current and next milestones.

Since we are async, we should not have to depend on a 'planning meeting' and instead, work on planning/reviewing issues and designs as they occur.  The managers of the team can serve as POs to make sure and remind the team we should be getting items planned early.  Ideally, we are planning/reviewing items for the next milestone, though as we start, there may be some items in the current milestones needing review.

cc @gitlab\-com/gl\-infra

Story points: 1.0