giseldo/neodataset huggingface dataset:
HAProxy alerts should fire based on error-rate rather than static value

From: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6397, we would like to alert based on error-rate rather than per-second average rate of increase of total errors. Alerts in the scope of this work: 

1. IncreasedBackendConnectionErrors <br/>
`rate(haproxy_backend_connection_errors_total[1m]) > .1`. What this means is that 0.1 * 60 = 6 error is enough to trigger this alert regardless of the total connections (which might also increase over time)
1. IncreasedServerConnectionErrors <br/>
`rate(haproxy_server_response_errors_total[1m]) > .5`. Here it means 0.5 * 60 = 30 error is enough to trigger this alert. 
1. IncreasedServerResponseErrors <br/>
`rate(haproxy_server_connection_errors_total[1m]) > .1`. Here, it is also 6 errors. 

The proposal here is to calculate the above based on error-rate where we calculate it as: `error count` / `total count`. For example, for the `IncreasedServerResponseErrors` and `api_rate_limit` backend we could do: 

`sum(rate(haproxy_server_response_errors_total{backend="api_rate_limit"}[1m]) * 60) / sum(rate(haproxy_server_http_responses_total{backend="api_rate_limit"}[1m]) * 60)`

We will then have to set a threshold of error rate. The highest number of requests `api_rate_limit` processed in the last 2 weeks was around 1600 requests per min (ref: https://prometheus.gprd.gitlab.net/graph?g0.range_input=1w&g0.expr=sum(rate(haproxy_server_http_responses_total%7Bbackend%3D%22api_rate_limit%22%7D%5B1m%5D))%20by%20(backend)&g0.tab=0). The current threshold of `> .1` would mean that 6 errors out of 1600 requests would be enough to trigger the alert and page us. This is 0.003%.

Story points: 4.0
Adjust HAProxy alert thresholds (for connection and response errors)

From https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6374, this issue will keep track of the work for: 

1. IncreasedBackendConnectionErrors
2. IncreasedServerConnectionErrors
3. IncreasedServerResponseErrors

Setting the threshold from 10seconds to 2mins. For reasons behind this proposal look at: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6374#note_151471405

Story points: 1.0
use disk snapshot to restore instances for testing database reviews

when we need to test some queries for database reviews, we have to restore the production wale base backup with the pipeline https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/pipelines which normally takes a few hours.

I propose we add a step to take snapshot of the boot disk and data disk in nightly restore schedule before deleting the restored instance. The snapshots can therefore be used to restore instances quickly to test queries for database reviews

Story points: 2.0
Rebuild DR File nodes with cheaper storage

File nodes are currently using a little over 500Tb of SSD storage.  Due to cost concerned, we need to move this to spinning disks.

Steps involved:

- [x] Tearing down and deleting the file nodes and their disks
- [x] Requesting a quota increase on spinning disks (and optionally a decrease on SSD’s)
- [x] Spinning up new instances with the disk type changed
- [ ] I will have to get some help from the Geo team on a clean way to reset the replication state, since replication has already started and is currently paused

Story points: 2.0
Lower the repository size limit on GitLab.com to 20GB

As part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5160 to reach the 10gb limit, let's:

* [x] Adjust the limit to 20GB 
* [x] Mail affected users of repositories above 20GB: https://gitlab.com/gitlab-com/marketing/digital-marketing-programs/issues/418

Story points: 2.0
New training GCP Project

We are planning a couple of workshops (https://gitlab.com/gitlabcontribute/new-orleans/issues/51 and https://gitlab.com/gitlabcontribute/new-orleans/issues/56) for Contribute 2019 and would like to use a dedicated GCP project where attendees can create/delete clusters.

We could name it `gitlab-training`.

Thanks

Story points: 1.0
Return corresponding error codes from haproxy

We should have error pages that report the appropriate error to the end-user instead of the same one for most 5xx codes:

```
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/429.http
    errorfile 408 /etc/haproxy/errors/400.http
    errorfile 429 /etc/haproxy/errors/429.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/500.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/500.http
```

Story points: 1.0
Database Reviews

* [x] @abrandl Rails 5.1 https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9386
* [x] @abrandl Int4Int8 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24512
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26009
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9939#note_150188904
* [x] @fjsanpedro -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25881#note_150153031
* [ ] @yguo -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26057#note_149890203
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25321#note_149349275
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2922/diffs#note_156988
* [x] DB office hours https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9861#note_152718524
* [x] DB office hours https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25973#note_152584625
* [ ] @yguo https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10331#note_152692635
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26146#note_152736732
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10234#note_152481170
* [ ] @cshobe https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9815#note_152275676
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10161#note_151035300

Story points: 5.0
CI/D Readiness Review Addendum Runbook Responses

In preparation for taking over the day to day of CI/CD issues, the runbooks should cover alerts that frequently have occurred.

- [ ] CPU use percent is extremely high on shared-runners-manager-4.gitlab.com for the past 2 hours.
- [ ] No disk space left on /opt/prometheus/prometheus/data on prometheus-01.nyc1.do.gitlab-runners.gitlab.net: 641.8m%
- [ ] No disk space left on /opt/gitlab on runners-cache-5.gitlab.com: 997.3m%
- [ ] Runners manager is down on shared-runners-manager-4.gitlab.com:9402
- [ ] [CICDTooManyPendingJobsPerNamespace](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L19-29)
- [ ] [CICDTooManyRunningJobsPerNamespaceOnSharedRunnersGitLabOrg](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L43-53)
- [ ] [CICDNamespaceWithConstantNumberOfLongRunningRepeatedJobs](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L210-223)
- [ ] [CICDJobQueueDurationUnderperformant](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L225-244)
- [ ] [CICDTooManyPendingBuildsOnSharedRunnerProject](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L4-17)
- [ ] [CICDTooManyArchivingTraceFailures](https://gitlab.com/gitlab-com/runbooks/blob/master/rules/gitlab-com-ci.yml#L263-275)

Story points: 1.0
Raise package upload size limit on packagecloud

Our nightly packages are now above 600MB which is the current `client_max_body_size`. We need to raise this higher to enable package uploads. However, we must wait until gitlab-com/gl-infra/infrastructure#6337 is complete as we will be required to restart packagecloud to make the change.

- [job failure](https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/106534)
- [Slack thread](https://gitlab.slack.com/archives/C1FCTU4BE/p1552062978274100)

Story points: 1.0
Run OPTIMIZE TABLE on packagecloud databases

Now that we are finally able to clean out all these extra rows in the metadata tables, we will need to run `OPTIMIZE TABLE` on `rpm_files` and `deb_files` in order to reclaim the disk space. This will lock the tables during its run, which means we will not be able to upload packages during that time, so should coordinate to ensure that it will not block package releases. It will most likely take around 10 minutes to complete this as our tables are now substantially smaller than they were when we did the upgrade.

Story points: 1.0
Update packagecloud backups to use xbstream

With the upgrade to 2.0.5, we are now able to use `xbstream` to backup the MySQL database. This will improve the speed and reliability of our backups and eventually assist us in moving to use RDS for the backend.

https://gitlab.com/gitlab-cookbooks/gitlab-packagecloud/merge_requests/13 will enable this in the config, but we need to wait for gitlab-com/gl-infra/infrastructure#6337 to complete first.

Story points: 1.0
Cleanup file metadata for packagecloud

## Summary

In packagecloud 2.0.5, they've added the ability to permanently disable the repo file list metadata, resulting in far fewer rows being added to the files tables, and kicking off a background job to cleanup the existing files rows for the repo you turn it on for. See https://packagecloud.atlassian.net/wiki/spaces/ENTERPRISE/pages/465076277/RPM+and+Debian+file+list+metadata for details on the new feature.

## Background

Packagecloud by default keeps track of every file within a package is a new role in one of it's files tables in the database, and uses this information to generate repo file list metadata files. Due to number of files in our omnibus package, the size of these tables is large, and is impacting our ability to backup the database. In prior versions of packagecloud we have turned off the metadata file generation, but this does not change the number of rows added to the database.


## Additional Info

This is some related information we recently got from our contact at packagecloud regarding turning off these files:

> Once you enable this setting, it will kick off a job that will delete all the unnecessary files rows in the database (this job will likely take a few days to run, but nothing will be locked in the process). Note: As the name suggests, this is a *permanent* setting, once filelists are disabled permanently, they can't be re-enabled. 

> (Optional) After doing the above and ensuring all of the DeleteFilesJob jobs are finished (you can see job progress in the indexer status section of the administrator dashboard) you can now run 'OPTIMIZE TABLE deb_files;' and 'OPTIMIZE TABLE rpm_files;' to significantly reduce disk space used and the size of your database backups.


## Actions

- [x] Permanent disable metadata for `gitlab/pre-release` (~ 10000 packages)
  * [x] DeleteFilesJob complete
- [x] Permanent disable metadata for `gitlab/nightly-builds` (~ 30000 packages)
  * [x] DeleteFilesJob complete
- [x] Permanent disable metadata for `gitlab/raspberry-pi2` (~ 1000 packages)
  * [x] DeleteFilesJob complete
- [x] Permanent disable metadata for `gitlab/unstable` (~ 6000 packages)
  * [x] DeleteFilesJob complete
- [x] Permanent disable metadata for `gitlab/gitlab-ee` (~ 5000 packages)
  * [x] DeleteFilesJob complete
- [x] Permanent disable metadata for `gitlab/gitlab-ce` (~ 5000 packages)
  * [x] DeleteFilesJob complete 
- [x] Run `OPTIMIZE TABLE deb_files;` (note table will be locked, uploads will be blocked)
- [x] Run `OPTIMIZE TABLE rpm_files;` (note table will be locked, uploads will be blocked)

Story points: 2.0
Look into set of old gsrm runners in gitlab-ci gcp project

There are a batch of runners still running since Feb 6 in the gitlab-ci project.

https://console.cloud.google.com/compute/instances?project=gitlab-ci-155816&instancessize=50&instancessort=creationTimestamp%252Cname%252CmachineType&instancesquery=%255B%257B_22k_22_3A_22_22_2C_22t_22_3A10_2C_22v_22_3A_22_5C_22gsrm_5C_22_22%257D%255D

We should look into if they are still running jobs, what happened and make sure they are properly destroyed.

cc @northrup as we were looking at usage on runners and this is an anomaly maybe?

8vcpu * 86400s = 691,000s per day

Story points: 2.0
DR Site Codebase out of sync with primary production site

The Geo nodes in DR are running `v11.6.3` - the production nodes are running `v11.8.1`

We need to get these running the same version before activating the DR site.

This should just be a matter of triggering the pipeline and making sure it succeeds.  If I can get some quick instruction on the proper way to do this, I'll make sure it gets into the DR runbooks.

Story points: 1.0
chef-client failing in dr pubsub because of ChecksumMismatch

Below the error:

```
remote_file[/opt/pubsubbeat/pubsubbeat] action create[2019-03-06T01:40:46+00:00] INFO: Processing remote_file[/opt/pubsubbeat/pubsubbeat] action create (gitlab-elk::pubsubbeat line 20)


    ================================================================================
    Error executing action `create` on resource 'remote_file[/opt/pubsubbeat/pubsubbeat]'
    ================================================================================

    Chef::Exceptions::ChecksumMismatch
    ----------------------------------
    Checksum on resource (b3b911) does not match checksum on content (2bb63f)

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/gitlab-elk/recipes/pubsubbeat.rb

     20: remote_file node["gitlab-elk"]["pubsubbeat"]["bin"] do
     21:   source node["gitlab-elk"]["pubsubbeat"]["url"]
     22:   checksum node["gitlab-elk"]["pubsubbeat"]["sha256sum"]
     23:   owner node["gitlab-elk"]["pubsubbeat"]["user"]
     24:   group node["gitlab-elk"]["pubsubbeat"]["group"]
     25:   mode "0755"
     26:   force_unlink true
     27:   notifies :restart, "runit_service[pubsubbeat]", :delayed
     28: end
     29:

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/gitlab-elk/recipes/pubsubbeat.rb:20:in `from_file'

    remote_file("/opt/pubsubbeat/pubsubbeat") do
      provider Chef::Provider::RemoteFile
      action [:create]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      source ["https://ops.gitlab.net/gl-infra/pubsubbeat/-/jobs/377/artifacts/raw/build/pubsubbeat"]
      use_etag true
      use_last_modified true
      declared_type :remote_file
      cookbook_name "gitlab-elk"
      recipe_name "pubsubbeat"
      checksum "b3b911b5802da256a379d1ad9ee65f3711c7b06e36181124697f824041cfaee1"
      owner "root"
      group "root"
      mode "0755"
      force_unlink true
      path "/opt/pubsubbeat/pubsubbeat"
      verifications []
    end

    System Info:
    ------------
    chef_version=12.22.5
    platform=ubuntu
    platform_version=16.04
    ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
    program_name=chef-client worker: ppid=15468;start=01:40:11;
    executable=/opt/chef/bin/chef-client
```

This should be fixed to ensure logging in Geo is working. See https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6330

Story points: 2.0
Move license-gitlab-com and version-gitlab-com from -com to -org

### Overview

We'd like to move the `license-gitlab-com` and `version-gitlab-com` project under the `gitlab-org` group. 

The Fulfillment team works on these two apps, and issues/boards aren't visible across non-nested groups. As a result, there is no single source of truth that gives an accurate assessment of everything that Fulfillment is working on in a single release. 

Deliverables may slip for this reason, simply because we need to remember to check issues in several different places.

* [x]  Move `gitlab-com/version-gitlab-com` into the `gitlab-org` group namespace
* [x]  Move `gitlab-com/license-gitlab-com` into the `gitlab-org` group namespace

Story points: 2.0
Upgrade kernels on API fleet only

Most of our production alerts are caused by random API server reboots. The current theory is that it is caused by NFS issues leading to a kernel panic and can be fixed by a kernel upgrade. As sequentially rebooting API servers can be done easily without customer impact (after draining from the LB) we should make this a priority before working on a framework to safely upgrade the kernels of the _whole_ fleet.

Story points: 3.0
Make rackspace users for infra team and other items

We only have rackspace users for part of the infra team so this issue will track:

1. [x]  Making the users - list below
2. [x] Updating SRE entitlements access template
3. [x] Make sure onboarding template verifies access.
4. [x] Double check the runbook is up to date:  https://gitlab.com/gitlab-com/runbooks/blob/master/howto/GCP-rackspace-support.md
5. [x] Broadcast to the team about creds and support tickets.

* [x] Amar
* [x] Ahmad
* [x] Jarv
* [x] Michal
* [x] Hendrik
* [x] Henri
* [x] Andreas
* [x] Jose
* [x] Skarbek
* [x] Yun
* [x] Cameron
* [x] Alejandro
* [x] Alex
* [x] Casey
* [x] Anthony
* [x] Craig
* [x] Devin

Story points: 2.0
Make the High4xxRateForRegistry alert less sensible in staging

The `High4xxRateForRegistry` is triggered by when 75% of the responses return a `4xx`:

```sum(backend_code:haproxy_server_http_responses_total:irate1m{backend="registry",code="4xx",tier="lb"}) / sum(backend_code:haproxy_server_http_responses_total:irate1m{backend="registry",tier="lb"}) > 0.75```

On staging we nearly have no requests at all leading to an alert as soon as only a few 4xx responses are triggered. This happened on 2019-03-04 13:16 UTC caused by a slow scan for non-existing endpoints, causing alerts for several hours.

https://gitlab.slack.com/archives/C101F3796/p1551702532086700

We should maybe add a minimum threshold of `x` requests/s before we trigger alerts besed on error rates on low-traffic servers.

Story points: 2.0
Upgrade PackageCloud

We have run into an issue where the RPM file table of the PackageCloud database is using signed 32bit integers for the `id` column and we've run into the limit. We've worked around the issue by keeping the rails app from updating that table, however without this table, it is possible that some RPM based package managers will have issues installing packages.

There is a new version of PackageCloud that will resolve this issue. This upgrade requires downtime. We have a call with PackageCloud to discuss the details of the upgrade.

- [upgrade docs](https://packagecloud.atlassian.net/wiki/spaces/ENTERPRISE/pages/465141825/Upgrading+from+2.0.4+to+2.05)
- [disable metadata table docs](https://packagecloud.atlassian.net/wiki/spaces/ENTERPRISE/pages/465076277/RPM+and+Debian+file+list+metadata) 


cc/ @twk3 @marin @rspeicher @dawsmith

Story points: 8.0
Cannot connect to port 80 on new AWS instance

> Hi, do you have any idea on why I can't access 80 port of i-02ad02527d4758cb3 instance on AWS. I checked security groups of the instance, Network ACL for subnet and for VPC. I also checked route table. 22 port works great

```
$ netstat -plnt
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      -
```

and

```
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
```

![Screen_Shot_2019-02-28_at_20.30.57](/uploads/202c4b9683caadb56c26b4523f03ebdf/Screen_Shot_2019-02-28_at_20.30.57.png)

Story points: 1.0
Clean up jobs.gitter.im Route53 record

Follow-up item per notes in gitlab-com/gl-infra/infrastructure#5516; we need to remove the `jobs.gitter.im` Route53 record from the [gitlab-com](https://gitlab-com.signin.aws.amazon.com/console) account, and do a quick review of the Ansible & Terraform codebase in gitlab-com/gl-infra/gitter-infrastructure> to verify that there are no additional impacts/changes required.

/cc @MadLittleMods in case I missed any details/context

Story points: 1.0
301 redirect codefresh page

Add a 301 redirect from `/devops-tools/codefresh/` to `/devops-tools/codefresh-vs-gitlab.html`

We get an inbound like from this [Codefresh blog post](https://codefresh.io/continuous-integration/codefresh-versus-gitlabci/) but it goes to a helper page we use to generate the real comparison page. 

Logging an issue here since we don't have https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4894 yet.

cc @brendan @shanerice 

Story points: 1.0
New GCP Project for GitLab PaaS POC/testing/alpha features enablement

The Configure group is currently working towards a POC for [GitLab PaaS](https://gitlab.com/groups/gitlab-org/-/epics/111). We'd like to create a dedicated project on GCP so that we can:

1. Enable GCP alpha features, namely [GKE Sandbox](https://cloud.google.com/blog/products/containers-kubernetes/digging-into-kubernetes-1-12)
1. Ensure we have IP space available (currently all other projects run out of IP space often, this would block us)
1. Analyze cost easily without having to filter resources out of existing projects

We only require that the project have billing enabled and have a recognizable name, such as `gitlab-paas`.

Also, if possible, it would be great to limit access to this project to members of the configure team (perhaps just for resource creation).

```
tkuah
dgriffith
jfargher
twatson
dgruesso
tdavis
mcabrera
ddavison
jcunha
jerasmus
gbizon
mgreiling
```

Thank you

Story points: 2.0
No Registry logs in Kibana

I wonder if this is because the timestamp is preceding the JSON logs?

```
2019-02-27_08:28:16.46583 {"go.version":"go1.10.3","http.request.host":"registry.gitlab.com","http.request.id":"09b57461-9cd7-4053-a7fa-c920e7e54a11",...
```

Story points: 2.0
Staging: delayed replica unable to follow timeline

The delayed replica in staging is unable to recover further:

```
2019-02-26_13:17:54.62610 2019-02-26 13:17:54 GMT [19449]: [192257-1] LOG:  new timeline 71 forked off current database system timeline 69 before current recovery point 36A6/D0000000
```

Additionally, we're getting paged on the production pagerduty schedule. I don't know if this is related or not, but it's not intended since this is about staging.

Example alert: https://gitlab.pagerduty.com/incidents/PGURCPE
Slack: https://gitlab.slack.com/archives/CB3LSMEJV/p1551186785243800

cc @yguo @cshobe

Story points: 2.0
ZFS: Establish QA Harness for Packer Image Build

QA Harness for Packer Builds Should:

1. [x] Validate that the image is bootable
1. [ ] Validate that the image can be bootstrapped into Chef
1. [ ] Validate that the entire GitLab Suite Functions and passes existing QA harness
1. [ ] Promote the image into selected train (i.e. current or next)

Stretch Goal: 
- [ ] Validate that the minimum convergence time between first spin to functional machine is < 5m

Story points: 8.0
ZFS: Establish Packer Build

Establish a CI process that meets the following criteria:

- [x] Starts with Google Ubuntu 18.04 LTS [Shielded VM](https://cloud.google.com/security/shielded-cloud/shielded-vm) Image
- [x] Leverages all of the cookbooks in the `gprd-base` role via a `chef zero` run process
- [x] Installs and configures ZFS kernel modules and supporting tools via `chef zero` run process
- [x] Produces a GCP Disk Image as the end product

Story points: 20.0
Staging alerts mis-labeled and going to production pager duty

The following alerts have been going to the production, high priority channel of pager duty all weekend:

```
Postgres Replication lag is over 9 hours on delayed replica (normal is 8 hours)
PostgreSQL_ReplicationLagTooLarge_ArchiveReplica
PostgreSQL_UnusedReplicationSlot
PatroniIsDown
```

All of these have hostnames in staging.

Additionally, this ops alert is going to production, high priority channel but is clearly not high priority.

```
Deadman switch of `test_alert` (db/postgres) has expired
```

Story points: 2.0