giseldo/neodataset huggingface dataset:
301 Redirects for Comparison Pages

We'd like to move `/comparison` to `/devops-tools` 

This will require 217+ redirects in order to route existing pages to new urls. 

Here is a [spreadsheet of the required redirects with the source URL and the desired: https://docs.google.com/spreadsheets/d/17cU2VUlIIaw9LEU1VnPuNrA2uBos3UAD8CItZP9ESzo/edit#gid=0

### Blocking work 

Before we put redirects in place: 

https://gitlab.com/gitlab-com/www-gitlab-com/issues/3093 and https://gitlab.com/gitlab-com/www-gitlab-com/issues/3094 should be done first

### WARNINGS

NOTE: Not all URLs are a 1:1 swap of s/comparison/devops-tools. See the [redirect spreadsheet](https://docs.google.com/spreadsheets/d/17cU2VUlIIaw9LEU1VnPuNrA2uBos3UAD8CItZP9ESzo/edit#gid=0) for the correct urls.

NOTE: Any additional comparison pages that are created between now and the 301s going live will be need to added to the 301 redirect spreadsheet.  

NOTE: This the timing of this merge will need to be coordinated with the corresponding www-gitlab-com merge https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15469. 

- If 301 redirects merge too early, then web visitors will be sent to 404 pages that don't exist yet 
- If 301 redirects merge too late, then any existing links and google juice will go to 404 pages

cc @sytses @kuthiala @dangordon 

Story points: 1.0
[Design Document] Ephemeral Environments

This document describes using Ephemeral Environments to provide a location to test infrastructure and application components, and make it easier to manage GitLab.com

The document can be reviewed at: https://add-ephemeral-environments-design.about.gitlab.com/handbook/engineering/infrastructure/design/201810_environments.html

The merge request is at https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15450

Story points: 15.0
Enable CloudTrail for AWS

We should ensure that we have global trails enabled within CloudTrail for AWS API audit logging

This may require creating/configuring a new S3 bucket for the CloudTrail logs (if an appropriate bucket doesn't already exist), and [creating a default trail](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html#creating-a-trail-in-the-console); for this default trail, we want to ensure that it is applied to all regions (step 5).

Story points: 1.0
HTTP targets missing TLS upgrade

There are a few instances of HTTP endpoints not doing a redirect to https:

* [x] `http://contributors.gitlab.com`
* [x] `http://dashboards.gitlab.com`
* [x] `http://dashboards.gitlab.net`
* [x] `http://gitlab-examples.gitlab.io`
* [x] `http://gitlabhosted.com`
* [x] `http://gitlabhq.com`
* [x] `http://log.gitlap.com`
* [x] `http://pages.gitlab.io`
* [x] `http://performance.gitlab.net`

Story points: 5.0
[Design Document] Git Workflow for CI/CD

This document describes a Git Workflow to facilitate using CI/CD to deploy infrastructure and application components, and make it easier to manage GitLab.com

To summarize, the standard [GitLab flow](https://docs.gitlab.com/ee/workflow/gitlab_flow.html) workflow is not the desired model for managing GitLab.com and related resources. This document discusses how we can use a common git workflow with GitLab's CI/CD tool set to manage all GitLab.com resources (Terraform, chef, Kubernetes, Monitoring, Deployments, etc). 

The document can be reviewed at https://add-infra-git-workflow-design.about.gitlab.com/handbook/engineering/infrastructure/design/201810_git_workflow.html

The initial merge request is here: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/15393/diffs

It should should be considered in parallel with [Dogfooding CI/CD](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5231) #5231

Story points: 5.0
Renewal of 4 TLS certificates

On November 7th, 2018 we have 4 TLS certificates expiring:

* `*.helm-charts.win`
* `*.k8s-ft.win`
* `*.cloud-native.win`
* `*.separate-containers.party`

These certificates are used for GitLab Charts Review Apps as well as part of development envs for Distribution team. The current certificates are stored in the `Cloud Native` vault, would be great to replace them there.

AC - see if we can get these managed by sslmate too for automating renewals


Story points: 1.0
forum.gitlab.com - Transition Backups to Cloud Storage

### Plan of Action
The Discourse forum engine has the ability to use AWS S3 as a backup target instead of local disk for it's nightly backups.  We should:

- [x] Create an AWS S3 bucket for forum.gitlab.com backups
- [x] Configure forum.gitlab.com in the admin interface to use the S3 bucket
- [x] Verify that backups are being written to the bucket

---
### Original Issue Description
The forum service has it's own backup process, ownership of this service is lacking and the ability to restore at any given point is not documented.  Backups are stored on the disk of the host itself.  We'll want to probably ship those offsite for cold storage in the case of disaster recovery.

Story points: 2.0
Runbooks and monitoring for forum.gitlab.com

With #8202 underway, this issue gains specificity as an ultimate input to a corresponding production readiness review under gitlab-com/gl-infra/readiness> for migrating/launching the service in k8s on GKE; that work will incur corresponding changes and requirements to the specifics of how we monitor services in k8s, metrics available, norms established for managing logs, etc.

This issue now becomes focused on authoring the initial set of runbooks and updating relevant architecture pages in the handbook to properly and comprehensively document the infrastructure for the forum service.

Original description
```
We do not have appropriate monitoring of the forum service.

The only alerts we get are when someone notifies us about backups failing: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5214

Or when pingdom tells us it's down: https://gitlab.slack.com/archives/C101F3796/p1539262868000100

We don't have our standard set of prometheus metrics being collected on this box.  We are unable to proactively handle potential situations that arise.

Use this issue to discuss how we can better manage this box.
```

Story points: 5.0
Fastly potentially serving incorrect content to some users

## Discovery

An example URL provided to me for testing this:
https://about.gitlab.com/direction/secure/

A few slack conversations:
* https://gitlab.slack.com/archives/C101F3796/p1539191863000100 (unthreaded)
* https://gitlab.slack.com/conversation/C9WFMSDFF/p1539167836000100 (thread)

Some team members are reporting different versions of the site multiple hours after a successful deploy.  One user is reporting that he see's what's been deployed on their mobile, while an old version is still showing on their workstation :unamused: 

Using Fastly's Check Cache feature, I've discovered 2 endpoints that have differing versions of the same page using the example above.  Those in question are:
* `cache-tyo19931`
* `cache-nrt6138`

I've opened a support ticket with Fastly about this.  I hesitate to purge cache in case they are able to perform some troubleshooting.

As an FYI the backup time of the cache is configured to 1 hour.  Being that our deploy completed over 3 hours prior, Fastly really ought to have the latest version of our site.

## Support
Fastly Support ticket: http://fastly.zendesk.com/hc/requests/98067

Story points: 1.0
Onboarding architectural program

To improve onboarding for new SREs we should give https://about.gitlab.com/handbook/engineering/infrastructure/production-onboarding/ a facelift. There are some good pointers there already but there is a lot of detail lacking.

This issue will be used to collect some thoughts, before a handbook MR.

I would also like to expand onboarding to include some more general architecture overviews. This would include tracing different types of requests through the gitlab.com architecture.

Since this page serves as a documentation index that is geared towards new-hires the sections could be fleshed out a bit. I don't see this as a replacement for the onboarding template, but a way to augment it with more information that can also be useful to non SREs so they can see what the typical day-to-day is for the team.


## oncall overview

## Where to find things

* fleet information
* logs, what are we storing and where to find them
* monitoring and alert rules
* dashboards
* rails and database access

## terraform

* environments and state files
* shared configuration overview, variables, etc.

## chef

* updating a cookbook, how much is automated, cookbook versions
* updating a node attribute
* updating a secret

## Architecture overview of different types of requests

* https get request to gitlab.com
* public api
* https git
* ssh git
* repository overview
* pages request
* registry
* download project zip file
* download project archive
* request raw file
* cicd job

## releases

Distilled version of how releases work and how deployments work with chatops?

Story points: 4.0
evaluate stickiness settings on haproxy backends

Currently the haproxy backend configuration is such that every pool of servers is round-robin with the exception of websockets, ssh and pages.


* `websockets`
```
    balance roundrobin
    cookie _gitlab_session prefix nocache
```
* `ssh`
```
    balance source
    hash-type consistent
```
* `pages_http` and `pages_https`
```
    balance source
    hash-type consistent
```
The other backends:

* `api`: all of the `api-xx` nodes
* `https_git`: all of the `git-xx` nodes
* `web`: all of the `web-xx` nodes
* `canary_web`: all of the `web-cny-xx` nodes
* `registry`: all of the `registry-xx` nodes

Use roundrobin load balancing. As now we are starting to put some production traffic on canary we should probably consider making requests sticky so that clients do not potentially see multiple versions of the applications on a single page load.

For `web` and `canary_web` this would mean modifying our load balancing algorithm to use the session cookie:

```
    balance roundrobin
    cookie _gitlab_session prefix nocache
```

For the `api` this would mean modifying our load balancing algorithm to use the source ip:
```
    balance source
    hash-type consistent
```

but I wonder how smart this will be since most of our traffic is coming from the CICD. Until we are able to separate that traffic we should probably leave it roundrobin.

For the `registry` I'm not sure whether it matters much but we could also use source ip here as well:

```
    balance source
    hash-type consistent
```

Story points: 2.0
Users not receiving emails

We have received a couple of requests related to service desk not sending emails or users are not receiving emails.

Ticket: https://gitlab.zendesk.com/agent/tickets/105220

https://gitlab.com/gitlab-com/support-forum/issues/3984

Few of the discussion: https://gitlab.slack.com/archives/C4XFU81LG/p1539078943000100 (internal)

Can we please check and investigate this.

cc// @ahmadsherif

Story points: 1.0
Enable Git v2 over SSH on GitLab.com

Now that https://gitlab.com/gitlab-org/gitlab-ce/issues/46555 will land in production soon (in the next RC) it will work for Git HTTP clients that request v2, but not for Git over SSH.

In order to enable v2 in production, we'll need to set the following in `sshd_config`:

```
AcceptEnv GIT_PROTOCOL
```

Now, this could have security considerations, such as https://serverfault.com/questions/427522/why-is-acceptenv-considered-insecure

And from the man page of SSHD:

```
Be warned that some environment variables could be used to bypass restricted user environments. For this reason, care should be taken in the use of this directive. The default is not to accept any environment variables.
```

So I think we need a security evaluation before enabling this, although given that it's restricted to `GIT_PROTOCOL` I didn't see a big concern in this particular situation (pinging @kathyw so this can be evaluated by Security)

Also, it could be that we could enable this directly in Omnibus (or a setting to be made) - @marin would this be possible? I've only found https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/docker/assets/sshd_config which is Docker related.

A few facts about the Git protocol v2:

- It's opt-in only, so clients (GitLab users) that support it (using Git `v2.18.0` onwards) need to explicitily pass a configuration to enable it.
- Not all Git commands are on v2, some will still execute v1
- `GIT_PROTOCOL` is only evaluated for us if the string contains `version=2`, everything else is ignored and won't be passed to `git`, defaulting to `v1`. 

@jramsay this means that without us enabling this SSH config, people using `Git v2` wouldn't be able to use v2 over SSH (will use v1 instead), but could use v2 over HTTP on GitLab.com. In the docs we could point users on how to enable it for on-premises, unless we decide to configure this in Omnibus directly.

Git Protocol v2: https://github.com/git/git/blob/master/Documentation/technical/protocol-v2.txt

Man page for SSHD: https://linux.die.net/man/5/sshd_config

cc @jacobvosmaer\-gitlab @DouweM

Story points: 2.0
Setup Geo with ops.gitlab.net instance

Right now we don't have a regular deployment of Geo until https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4741. Could we do something simpler and just set up a single Geo instance for the ops instance and deploy regularly to it?

That way we can:

1. Ensure database migrations work
2. Ensure existing functionality still works
3. Test new functionality

Story points: 4.0
Unable to unarchive projects

A user is unable to unarchive these projects

```
https://gitlab.com/blade-group/dev/clients/boxes/audio-switcher 
https://gitlab.com/blade-group/dev/clients/boxes/electron
```

I believe this has something to do with our GCP migration. I was unable to reproduce this on my projects.

https://gitlab.zendesk.com/agent/tickets/105222

cc// @jarv

Story points: 1.0
Database Reviews

New issue for the current milestone that contains pending reviews from the last.

* [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2498/diffs
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7341#note_106524072
* [x] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2528#note_136727
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21719#note_104392420
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22014
* [x] Thursday https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7578#note_106070226
* [x] Thursday https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7806?commit_id=5d6177e91a8159dded6c5f62b5bd40a4165fe687#note_108912846
* [x] https://gitlab.com/gitlab-org/gitlab-ee/issues/6070#note_106815144
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22143#note_106796618
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6947
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22013#note_106869076
* [x] https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/678
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22226
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21021
* [x] https://gitlab.com/gitlab-org/gitlab-ee/issues/7955#note_109956059 together with https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7990#deca3eb05754841e761d010d8941933fd19f2aef_0_26
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22430

Story points: 5.0
Create a critical PD alert for HAProxy connection errors

corrective action for https://gitlab.com/gitlab-com/gl-infra/production/issues/496
Alert on https://dashboards.gitlab.com/d/ZOOh_aNik/haproxy?panelId=57&fullscreen&orgId=1&from=now-7d&to=now&refresh=5m&var-host=fe-07-lb-gprd.c.gitlab-production.internal&var-port=9101&var-backend=All&var-frontend=All&var-server=All&var-code=All&var-interval=30s as it is currently our best way to determine a malfunctioning backend server that is not accepting ssh but is passing the rails healthcheck.

Discussion about how to improve the healthcheck more generally in https://gitlab.com/gitlab-org/gitlab-shell/issues/166

Story points: 1.0
IAM policy gitlab-internal: For Configure team

Similar to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5233 I would like to request the same permissions for the following people:

- [ ] mcabrera@gitlab.com
- [ ] tkuah@gitlab.com
- [ ] taurie@gitlab.com
- [ ] mgreiling@gitlab.com
- [ ] dgruesso@gitlab.com
- [ ] jerasmus@gitlab.com

They require IAM permissions for `gitlab-internal-153318` project with role `roles/container.admin`.

Story points: 1.0
canary.gitlab.com should use the corresponding backends for api/git traffic

Our routing logic for canary is currently:

```
use_backend canary_web if is_canary or is_canary_host
```

Which means all requests to canary.gitlab.com or web requests that have the cookie set will go to the web fleet.
Ideally public api and git-ssh/git-https should use the new canary git and api servers.

Story points: 8.0
Convert environment labels to server-side

In order to improve metrics and alert routing and maintenance we should use Prometheus "external labels".

External labels allow Prometheus to assign labels based on its view of the system.

We're currently [populating several labels](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gprd-infra-prometheus-server.json#L6-9).
* `provider` (gcp/azure/etc)
* `region` (us-east, etc)
* `monitor` (default, app, etc)

We're currently overloading `monitor` with things like `gprd-default`.

I propose we change this to add a new `env` label that contains the environment for the Prometheus server.

This will allow us to transition alert routing to the new labels.

Story points: 3.0
Evaluate timed incremental rollout on hosting servers

We're going to ship the delayed job feature https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767 for AutoDevOps timed incremental rollout https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22023.

We're going to evaluate it works on our hosting servers with AutoDevOps timed incremental rollout mode.

### Evaluation plan

**dev.gitlab.org**

- Enabled term: 8th, Oct. ~
- Evaluation date: 8th, Oct.
- [x] Wait for dev.gitlab.org daily sync
- [x] Create a sample project with new AutoDevOps deployment strategy to make sure it's fully functional.
- [x] Check health (metrics, logs and crash reports)

**staging.gitlab.com**

- Enabled term: 10th?, Oct. ~ (It depends on RM's plan)
- Evaluation date: 10th, Oct.
- [x] RC with the new code has been deployed
- [x] Create a sample project with new AutoDevOps deployment strategy to make sure it's fully functional.
- [x] Check health (metrics, logs and crash reports)

**gitlab.com**

- Enabled term: 10th?, Oct. ~ (It depends on RM's plan)
- Evaluation date: 10th, Oct. ~ 10th, Nov.
- [x] RC with the new code has been deployed
- [x] Create a sample project with new AutoDevOps deployment strategy to make sure it's fully functional.
- [x] Check health (metrics, logs and crash reports)

NOTE: 
- This feature is behind the feature flag `ci_enable_scheduled_build`, but it's enabled **by default** (See https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767#note_106332776)
- This feature doesn't run unless users manually changed their deployment strategy in AutoDevOps. Presumably, after we published a release post of 11.4 at 22nd, Oct., users would try to use it, and the usage of the new sidekiq-workers will gradually increase so that the time we should keep eyes on server's health.

### Check health (metrics, logs and crash reports)

- Does the new worker `Ci::BuildScheduleWorker` run properly? This uses `pipeline_processing` namespace (priority: 5 (highest)).
- Does not the new worker `Ci::BuildScheduleWorker` pressurize other workers in the same namespace?
- Are there any stale delayed jobs? `Ci::Build.stale_schedule.count` should be zero.
- Are there any crash reports related to this feature on Sentry?
- StuckCIJobWorker should use Index Scan properly? (Ref: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767#note_106811708) (cc @abrandl)

### Feature flag

The feature flag's name is `ci_enable_scheduled_build`. The new AutoDevOps deployment strategy - Timed incremental rollout is based on the [delayed job](https://gitlab.com/gitlab-org/gitlab-ce/issues/51352) feature. By disabling `ci_enable_scheduled_build`, we can effectively revert the timed incremental rollout to manual incremental rollout (Also, stops new creation for delayed jobs)

```
Feature.enabled?('ci_enable_scheduled_build') # Check if it's enabled
Feature.enable('ci_enable_scheduled_build') # Enable the feature
Feature.disable('ci_enable_scheduled_build') # Disable the feature
```

/cc @nolith @winh @jlenny @erushton

Story points: 2.0
Out of disk space errors (inodes?) in artifacts tmp directory

Sentry error: https://sentry.gitlab.net/gitlab/gitlabcom/issues/530892

All the API servers are complaining about similar errors:

```
Errno::ENOSPC: No space left on device @ dir_s_mkdir - /var/opt/gitlab/gitlab-rails/shared/artifacts/tmp/cache/1538592524-29247-0008-8556
```

The disk is only 23% full. I suspect we are out of inodes in that parent directory.

Story points: 8.0
Move two projects from /jamedjo to /gitlab-com group

## What

Move https://gitlab.com/jamedjo/team-quiz and https://gitlab.com/jamedjo/icebreakers to the `gitlab-com` group

## Why

The [handbook says](https://about.gitlab.com/handbook/contracts/#approval-for-outside-projects) projects "completed as part of the GitLab employment" should be in a GitLab namespace, and I'd like to have the projects moved before announcing the gitlab-pages URLs

## Why in infrastructure

Changing group needs the Owner (project creator) to be a Maintainer in the target group. I thought I'd be able to get a maintainer to move it, but they wouldn't be the owner of the project. See https://gitlab.com/gitlab-org/gitlab-ce/issues/18423

Story points: 1.0
Rollout the Sidekiq Reliable fetcher

We have an issue for 11.4 https://gitlab.com/gitlab-org/gitlab-ee/issues/7279. It's already implemented and is expected to be reviewed and merged for 11.4.

By default, it's disabled and we can enable it with a feature flag `gitlab_sidekiq_reliable_fetcher`. The state of this flag is only considered when we run the Sidekiq process. So we need to restart the Sidekiq process on some node to enable it gradually and to see how it goes.

To monitor the state of the Sidekiq we should check the Grafana dashboards in https://dashboards.gitlab.net
and we can also see to the new "Working" tab in the admin area (Monitoring > Background jobs).

If something goes wrong we have to disable the feature flag and restart the Sidekiq process.

I'll also need some assistance from someone from the Production team

Story points: 1.0