giseldo/neodataset huggingface dataset:
RCA: API fleet unavailable 20181129 20:10:00 - 20:36:00 UTC

## Summary

All of the API servers became unavailable due to someone walking all of the public projects.

- Service(s) affected : API
- Team attribution : SAE
- Minutes downtime or degradation : 26 minutes

## Impact & Metrics

Start with the following:

- The API was completely inaccessible
- All users trying to access the API were affected
- The incident prevented any API actions, including pushes and CI runners fetching jobs 

### Graphs and logs

- [Kibana: Project listing requests](https://log.gitlab.net/goto/d1d21142d69d5ff6f30891a3cde44f4c)
- [Dashboards: DB host load and locks](https://dashboards.gitlab.net/d/000000142/postgresql-overview?orgId=1&from=1543521758415&to=1543524942335)
- [Dashboards: Workhorse Availability](https://dashboards.gitlab.net/d/OktWokpik/workhorse-overview?panelId=94&fullscreen&orgId=1&from=1543520954651&to=1543525648769)
- [Dashboards: HAProxy Dashboard during incident](https://dashboards.gitlab.net/d/ZOOh_aNik/haproxy?orgId=1&from=1543521006515&to=1543524880550&var-host=fe-01-lb-gprd.c.gitlab-production.internal&var-port=9101&var-backend=api&var-frontend=All&var-server=All&var-code=All&var-interval=30s)

## Detection & Response

- The incident was detected with a PagerDuty alert. The first alert resolved itself shortly after firing. The second one was the downtime.
- Remediation happened on its own.

## Timeline

2018-11-29

- 20:11 - First alert goes off `IncreasedBackendConnectionErrors`
- 20:16 - First alert clears
- 20:26 - Second alert fires `IncreasedBackendConnectionErrors`
- 20:29 - Users begin reporting issues
- 20:36 - The API servers recover and service is restored
- 20:41 - Second alert clears

## Root Cause Analysis

Why did the API servers become unavailable?
  - Public projects were being iterated through the API
  - API pagination issues can cause timeouts and overwhelm the servers (https://gitlab.com/gitlab-org/gitlab-ce/issues/42194)

## What went well

- We discovered the problem quickly and there was a quick response

## What can be improved

- This has happened multiple times (see https://gitlab.com/gitlab-com/gl-infra/production/issues/589)
- The runbooks for HAProxy problems could use some work
- We could further limit requests to this endpoint
- We can improve pagination in the application

## Corrective actions

- Dedicated fleet for internal API: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4708
- Better handling of pagination for api requests:  https://gitlab.com/gitlab-org/gitlab-ce/issues/42194

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 2.0
empty the artifacts bucket in GCS

We had a sync set up to sync artifacts from S3 to GCS nightly because we were hoping to migrate to GCS quickly at the time. In the meanwhile, we have encountered problems and the migration has been placed in the backlog (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4684#note_117887957 for more info). We need to empty out the artifacts bucket in GCS (`gitlab-gprd-artifacts`) and disable the nightly sync until we are closer to actually being able to execute the move. It is simply a massive waste of money to store all that data twice as well as bandwidth transfer costs with no migration time in sight.

cc/ @dawsmith @gitlab\-com/gl\-infra

EDIT:

As per [the object storage dashboard](https://dashboards.gitlab.net/d/BtEcmxLik/object-storage?orgId=1&from=now-12h&to=now) we are storing 307TB in GCS for artifacts. This amounts to 

```
314368 GB * $0.026/month = $8173.56/month
```

~~It isn't killing us, but it is also a waste.~~

EDIT2: On second thought, 8173.56/mo is a lot and could be an additional engineer.

Story points: 1.0
Readiness review for the new post deployment patcher

This issue is for an SRE review of the post deployment patcher.

- [x] SRE sign-off on new patcher tool https://ops.gitlab.net/gitlab-com/gl-infra/patcher
- [x] review new documentation - https://gitlab.com/gitlab-org/release/docs/merge_requests/84
- [x] deprecate old patcher howto - https://gitlab.com/gitlab-com/runbooks/merge_requests/840
- [x] deprecation notice on gitlab-patcher - https://gitlab.com/gitlab-com/gl-infra/gitlab-patcher/merge_requests/7
- [x] deprecation notice on post-deployment-patches - https://dev.gitlab.org/gitlab/post-deployment-patches/merge_requests/118
- [x] Turn off issues and MRs on the gitlab-patcher and post-deployment-patches repos
- [x] Apply the README update to point to official docs - https://ops.gitlab.net/gitlab-com/gl-infra/patcher/merge_requests/12
- [x] Deploy a noop patch (adds a comment to a source file) through the pipeline
- [x] Announce in `#backend` and to previous developers who have issued patches that the process has changed.

Story points: 2.0
Users unable to create project

Users are facing issues when they are trying to create a new project from web UI. They are getting:

`
PG::QueryCanceled: ERROR: canceling statement due to statement timeout CONTEXT: while rechecking updated tuple (1,14) in relation "site_statistics" : UPDATE "site_statistics" SET "repositories_count" = "repositories_count"+1
`

ZD: 
* https://gitlab.zendesk.com/agent/tickets/109256
* https://gitlab.zendesk.com/agent/tickets/109252

Story points: 1.0
The following servers should be added to blackbox monitoring for SSL certificate expiration

* `prometheus-01.nyc1.do.gitlab-runners.gitlab.net`
* `prometheus-01.us-east1-c.gce.gitlab-runners.gitlab.net`
* `prometheus-01.us-east1-d.gce.gitlab-runners.gitlab.net`

Reference: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5610

Story points: 1.0
updates on the monitoring script

1 - change the queries to check the database activity
select count(*) from pg_Stat_activity where state !='idle';

select state, count(*) from pg_Stat_activity group by state;

2 - create pre-flight checks to check the health status of the new cluster: 
-Pgbouncer
-Patroni 
-Postgresql

3 - Auto check results instead of human analysis.let's try to reduce.

4 - tmux topic... can we remove? and add the logging solution ? 

Story points: 8.0
problems with the Application LB

Problems in the tests with the application LB.

```
NoMethodError (undefined method `load_balancer' for nil:NilClass):
  ee/lib/gitlab/database/load_balancing/sticking.rb:81:in `load_balancer'
  ee/lib/gitlab/database/load_balancing/sticking.rb:34:in `all_caught_up?'
  ee/lib/gitlab/database/load_balancing/sticking.rb:42:in `unstick_or_continue_sticking'
  ee/lib/gitlab/database/load_balancing/rack_middleware.rb:21:in `stick_or_unstick'
  ee/lib/ee/api/helpers.rb:33:in `block in current_user'
  lib/gitlab/utils/strong_memoize.rb:26:in `strong_memoize'
  ee/lib/ee/api/helpers.rb:28:in `current_user'
  app/helpers/sentry_helper.rb:9:in `sentry_context'
  lib/api/helpers.rb:386:in `handle_api_exception'
  lib/api/api.rb:83:in `block in <class:API>'
  ee/lib/omni_auth/strategies/group_saml.rb:22:in `other_phase'
  lib/gitlab/middleware/multipart.rb:101:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  ee/lib/gitlab/jira/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/read_only/controller.rb:40:in `call'
  lib/gitlab/middleware/read_only.rb:16:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/request_context.rb:20:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
  lib/gitlab/middleware/release_env.rb:10:in `call'
```

Story points: 2.0
Disable Geo on staging

I just noticed Geo is still enabled on staging.

```ruby
GeoNode.secondary_nodes
=> #<ActiveRecord::Relation [#<GeoNode id: 100008, primary: false, oauth_application_id: 110492, enabled: true, ... url: "https://azure.gitlab.com/", selective_sync_type: "", selective_sync_shards: [], verification_max_capacity: 100>]>
```

This is a leftover from the GCP migration rehearsals and I don't see any reason why we wouldn't disable Geo completely on staging.

#### Backstory

During the rehearsals we did not disable Geo, but added a dummy node to the admin dashboard. This would maintain the Geo event log, which _might_ come in useful if we ever wanted to fallback.

Story points: 2.0
Permission request: role Kubernetes Engine admin on gitlab-internal project

Hi,

I'm trying to create a kubernetes cluster in the gitlab-internal project for me to test with, but I get the following error:

>  clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "rpereira@gitlab.com" cannot create clusterrolebindings.rbac.authorization.k8s.io at the cluster scope: Required "container.clusterRoleBindings.create" permission.

Can I be given the required permission/role (Kubernetes Engine Admin possibly)?

cc @northrup

Story points: 1.0
Renew certificates for CI Prometheus servers fleet

Certificates for CI Prometheus servers have expired at November 7th :(

Please renew them (they should be managed by sslmate) and update proper vaults:

- [x] `prometheus-01.nyc1.do.gitlab-runners.gitlab.net` (vault: `gitlab-runners-prometheus-do-nyc1 ci-prd`)
- [x] `prometheus-01.us-east1-c.gce.gitlab-runners.gitlab.net` (vault: `gitlab-runners-prometheus-gce-us-east1-c ci-prd`)
- [x] `prometheus-01.us-east1-d.gce.gitlab-runners.gitlab.net` (vault: `gitlab-runners-prometheus-gce-us-east1-d ci-prd`)

Story points: 1.0
Runbook and alerting improvements for staging.gitlab.com

As part of https://gitlab.com/gitlab-com/gl-infra/production/issues/586, we need to take the below corrective action items:

- [x] Improve runbook with more troubleshooting tips 
- [x] Create alerts on `node_netstat_TcpExt_ListenDrops` and/or `node_netstat_TcpExt_ListenOverflows`

Story points: 1.0
Cleanup and unify postgres dashboards

The Grafana dashboards for postgres are inconsistent in their use of variables and have hardcoded values for environment, type and prometheus instances in many places, which makes it hard to adapt them for new clusters or envs (like for patroni).

We should update the dashboards to consistently use selectable variables for `environment` and add a new variable `type` to switch between "patroni", "postgres" and possibly other db host types in the future. That would make the dashboards re-usable when adding patroni to production.

Story points: 1.0
301 Redirect for UX Research Panel

Please redirect: https://about.gitlab.com/researchpanel/ to https://about.gitlab.com/community/gitlab-first-look/

Please note a redirect is already in place from https://about.gitlab.com/researchpanel/ to https://about.gitlab.com/community/researchpanel/ This should be replaced with the above.

cc @williamchia

Story points: 1.0
Use ElasticSearch Curator for recurring ElasticSearch maintenance tasks

The [Curator tool](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) completes regular maintenance tasks such as trimming indices, purging data etc by using an easy to configure YAML definition file. Although the activities we currently do can (and are) simply completed with a bash script, the same activities can be done with Curator. The Curator also offers additional functionality of course. Once we upgrade to ES 6.5, the current bash maintenance script won't be necessary (AFAIK) as the functionality is included in ES 6.5. However, Curator may still offer functionality that would be relevant to maintenance of ES.

Story points: 1.0
Upgrade of Elastic.co hosted ElasticSearch from 5.6.10 to 6.5.1

ElasticSearch version 6.x has been available for a while and is supported by Elastic.co. GitLab should upgrade to take advantage of the latest features and speed improvements. It is also highly recommended by our Elastic.co technical contacts that we upgrade as soon as it is possible. 

The upgrade will be automatically handled by Elastic.co, and should be frictionless. However, we need to be aware of any potential changes to underlying attributes/fields. Breaking changes are described here: https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes.html

The upgrade will be a rolling upgrade as described here: https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html. The upgrade will require a full cluster restart which would cause the cluster to be unavailable for a short period of time.

To complete this tasks, I believe we will need to 

* [ ] Determine best approach for testing the upgraded version
* [ ] Create a new cluster based on latest version of ES
* [ ] Re-point some data ingestion points, potentially for staging, to new cluster
* [ ] Ensure no errors and data is available
* [ ] Remove re-point
* [ ] Upgrade production cluster
* [ ] Hope for the best

The disadvantages of this approach is the full cluster restart and thereby losing logs for a short period of time, which may be acceptable. 

I don't know the configuration/environment well enough to determine if the above steps are sufficient or even doable.

Story points: 3.0
Marvin down, returns a bundler error

# Summary

The marvin bot on slack is no longer responding to commands but is only showing the error below: 

```
/usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:172:in `user_home': uninitialized constant #<Class:Bundler>::Etc (NameError)
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:197:in`user_bundle_path'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler/settings.rb:377:in `global_config_file'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler/settings.rb:80:in`initialize'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:257:in `new'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:257:in`settings'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:84:in `configured_bundle_path'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:335:in`use_system_gems?'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:519:in `configure_gem_path'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:512:in`configure_gem_home_and_path'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:66:in `configure'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:134:in`definition'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler.rb:101:in `setup'
   from /usr/lib/ruby/gems/2.5.0/gems/bundler-1.16.2/lib/bundler/setup.rb:20:in`<top (required)>'
   from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:135:in `require'
   from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:135:in`rescue in require'
   from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39:in `require'
   from /home/bundle/cog-command:7:in`<main>'
exit status 1
```

Story points: 1.0
sentry-01-sv-ops.c.gitlab-ops.interal

```
  ================================================================================
  Recipe Compile Error
  ================================================================================

  Chef::Exceptions::RecipeNotFound
  --------------------------------
  could not find recipe server for cookbook postgresql

  Cookbook Trace:
  ---------------
    /var/chef/cache/cookbooks/gitlab-sentry/recipes/default.rb:9:in `from_file'

  Relevant File Content:
  ----------------------
  /var/chef/cache/cookbooks/gitlab-sentry/recipes/default.rb:

    2:  # Cookbook Name:: gitlab-sentry
    3:  # Recipe:: default
    4:  #
    5:  # Copyright 2016, GitLab Inc.
    6:  #
    7:  # All rights reserved - Do Not Redistribute
    8:  #
    9>> include_recipe 'postgresql::server'
   10:
   11:  # Fetch secrets
   12:  sentry_secrets = get_secrets(node['gitlab-sentry']['secrets']['backend'],
   13:    node['gitlab-sentry']['secrets']['path'],
   14:    node['gitlab-sentry']['secrets']['key'])
   15:
   16:  sentry_conf = Chef::Mixin::DeepMerge.deep_merge(sentry_secrets['gitlab-sentry'], node['gitlab-sentry'].to_hash)
   17:
   18:  ## nginx
```

Story points: 1.0
Load Balancer Redundancy Threshold too high

We get FeLoadBalancerLossOfRedundancy alerts to pagers any time there is ANY loss of redundancy in the load balancers.  While we want to know if this happens, that should be more of an informational thing until we get below about 80% to 90% redundancy.  What are the load balancers for, if not to automate managing our redundancy for us.

We should be able to lose some of our redundancy in the load balancer without setting off pagers.  To implement this, I've changed the threshold from 100% to 90% in https://gitlab.com/gitlab-com/runbooks/merge_requests/832

@gitlab\-com/gl\-infra - Please discuss if this is the appropriate threshold or what might be better.

Story points: 1.0
Create an alert when we are missing haproxy logs

It's happened too many times.  Our logs have disappeared from stackdriver.  Whether it be an incorrectly configured exclusion rule, or a misconfigured td-agent.  Both of these have negatively impacted our visibility into our environment.  Let's prevent this from happening by somehow alerting when we don't have logs available.

Acceptance Criterion:
* Determine a method for which we can alert on this
* Determine the appropriate threshold (if necessary)

Story points: 2.0
Handbook update to include a service list with owners and responsible

As part of the onboarding, we have identified a potential improvement which is a list of services that are currently known (and potentially managed by gl-infra). This update should be in the Handbook.

Some of the information is being captured in the Google Doc https://docs.google.com/document/d/1xxE-dZIArF59MMH0pbrnjRz3KlrtoNUOKnDwbf7N4L0. Once completed, the essential information must be extracted and added to the handbook.

Potentially a table with some of the following:
* Service URL
* Description
* Owner
* Responsible team
* Any potential documentation

Story points: 1.0
Streamline service access requests with gl-security Access Requests

Streamlining the access requests to be provisioned by gl-infra with the gl-security access request model would ensure compliance while simplifying the process flow. The idea is to ensure all access requests are managed using https://gitlab.com/gitlab-com/access-requests/issues but any requests requiring gl-infra to provision are passed to gl-infra in a suitable and timely manner. The issue is about creating the proper process or technical flow to ensure this is achieved.

We would like to avoid access requests being created directly in the gl-infra issue tracker. E.g. avoiding this [https://about.gitlab.com/handbook/engineering/#amazon-web-services-aws](https://about.gitlab.com/handbook/engineering/#amazon-web-services-aws)

Tasks:
* [ ] Agree appropriate model with gl-security
* [ ] Update access request template
* [ ] Update Security owned application list with agreements
* [ ] Update handbook with appropriate links

Story points: 1.0
Refactor common config in fluentd templates

There is a lot of copy and paste in the fluentd templates that can be refactored using includes.
It was also noticed that the prometheus mixin is missing in the redis config https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/merge_requests/55#note_119012949

This issue is to pull out as much of the common config as we can to make these templates a bit more manageable.

https://gitlab.com/gitlab-cookbooks/gitlab_fluentd

Story points: 2.0
Adjust disk capacity for postgres

Currently, we provision 5 TB of SSD disk space per database instance in both `gstg` and `gprd`.

Actual current usage is:
* `gstg`: 0.8 TB
* `gprd`: 2.3 TB

Leaving room for WAL and still [maxing out IO performance](https://cloud.google.com/compute/docs/disks/performance#size_price_performance), the proposal here is to reduce the disk capacity to:

* `gstg`: 1.5 TB - utilization approx. 54 %
* `gprd`: 4 TB - utilization approx. 58 %

We have 8 instances per environment. The saving in `gstg` is 28 TB and in `gprd` 8 TB.

This translates to a saving of roughly 6,300 USD per month.

Performance-wise, this does *not* impact `gprd`. For `gstg`, IO performance this change reduces sustained read performance to roughly half of what we currently have. However, we also have smaller instance types in `gstg` anyways.

Worth to note that increasing a disk is easy and does not require a reboot.

Story points: 2.0
Review staging env / cookbooks for patroni

Please consider and review the maintenance plan.

Notes:

* [ ] Disk size is 5TB, probably 4TB (in gprd) / 1.5TB (in gstg) is more than enough and we save $$$. Check performance implications. -> https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5566 -> https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/632
* [x] Compile a postgres-config diff between gprd and gprd-patroni before it goes live and make sure we're aware of all changes.
* [ ] Do we still need to manage `shmall` and `shmmax` ourselves or is this a relict? AFAIK it was only necessary in older postgres versions. 
* [x] Increase disk size for root volume `/` to 100 GB. -> https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/632

Story points: 3.0
Clean up old S3 registry bucket

With the successful migration of the registry to GCS (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4710), we need to look into cleaning out and removing the old bucket.

The size of the registry bucket is:

```
Selection: 0 Objects, 1 Folders Total size: 41.8 TB Total objects: 2983000
```

Story points: 3.0
Database Reviews

* [x] Sean https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8070
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22971#note_118167275
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23058
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23098#note_118544178
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8446#note_118367363
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23057#note_118367296
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23098#note_119437237
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23314/diffs#note_119525749
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23236#note_119675631
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19740#note_120436389
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22744
* [x] Andreas https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19740#note_120734824
* [x] Fri https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6878
* [x] Fri https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23412

Story points: 8.0
Infra Team Hiring/Questionnaire grading for Nov 18 - Dec 1

Team bucket to account for effort into hiring.
Questionnaires and interview time.

Starting with 13 Questionnaires to do.

The candidates to be assessed can be found here: https://app2.greenhouse.io/plans/4050457002/candidates?hiring_plan_id%5B%5D=4050457002&job_status=open&sort=last_activity+asc&stage_status_id%5B%5D=4000006002&type=all&in_stages%5B%5D=Assessment


List reviewed 20 of November :

Pending tickets : 
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5569
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5570

Story points: 13.0