giseldo/neodataset huggingface dataset:
Incident Slack Tooling

**------------------**<br>
**Update 2019-12-06**

We created this epic: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/100 to keep track of the overall incident management automation effort which originated from this issue. Keeping track of the effort under an epic made more sense since the subtasks would enumerate pretty quickly. Therefore, closing this issue. <br>
**------------------**

In order to streamline incident handling, we must **reduce cognitive load** on the Incident Team.

One way to do so is to ensure procedures are as simple as possible so that the Incident Team, and, in particular, the `EOC` and the `IMOC`, can focus on the technical resolution of the incident instead of the necessary procedural calls that are intended to provide visibility into and awareness of the incident.

Another way is to improve automation to support said procedures. In particular:

* **incident declaration**: 
  * opening `production` issue, 
  * creating tracking document, and
  * creating `infrastructure` root-cause analysis issue
  * optionally, severity assignment (and possibly `IMOC` escalation) and Security escalation (in cases of abuse) to notify the Security team
* **incident management**:
  * editing incident severity (and possibly `IMOC` escalation)
  * Security escalation
* **incident data management**:
  * ensure incident issues have severity labels
  * ensure incident issues have service labels
  * ensure incident issues have attribution labels
  * ensure incident issues have timely RCAs 

## Samples

As I am not entirely familiar with Slack's custom command capabilities, the following examples are provided as a CLI implementation:

#### Incident Declaration and Resolution

```
> incident declare [--abuse] <severity> [@user] <incident description>
```

* Creates a tracking document in Google docs from a template (ideally filling out some data)
* Creates an **incident** issue in the `production` queue and assigns it to *\@user*; if user is not specified, it is assumed to be the `EOC`
* Creates a corresponding issues in the `infrastructure` queue and links it to both the incident issue and the tracking document; assigned to EOC.
* When the `--abuse` flag is used, Security on-call is paged through PagerDuty

This command should return some way (an `<id>`) to refer to the incident from Slack. In commands that would normally use the `<id>`, when omitted, it is assumed to be the currently opened incident.

```
> incident resolve [<id>]
```

Resolves the incident by closing `production` incident issue.

#### Incident Management

```
> incident list
```

Lists open, on-going incidents, if any.

```
> incident status
```

Provides status on current/last incident (open, closed)

```
> incident <id> severity <severity>
```

Changes incident severity to `<severity>`; when severity is `S1` or `S2`, escalate to `IMOC`.

#### Incident Data Management

A bot that ensures incident issues have proper labels.

Story points: 2.0
Move all pingdom checks over to runbook configuration

Currently, a subset of pingdom checks are managed as YAML. See https://gitlab.com/gitlab-com/runbooks/blob/master/pingdom/pingdom.yml

Now that we're confident in this approach, we should bring them all in so that they're all managed through the YAML configuration. 

Related slack thread: https://gitlab.slack.com/archives/C101F3796/p1542354588701400

cc @jarv

Story points: 2.0
template for Patroni failover - postgresql steps

Please review the MR : 
https://gitlab.com/gitlab-com/migration/merge_requests/202

The repo is : git@gitlab.com:gitlab-com/migration.git

Create the steps for the maintenance for the Patroni failover.
Postgresql / Repmgr / Patroni steps to setupthe new cluster to production.

Please everything automated. Won't be allowed any direct interaction on hosts. All actions from the bastions.

Consider also to update/clean up the monitoring to do not consider REPMGR the master cluster.

@ahmadsherif  / Ongres @3manuek  are working on the postgresql.

@glopezfernandez @andrewn @dawsmith  @3manuek  @abel3  @ahmadsherif fyi

Story points: 8.0
template for Failover to Patroni

Please review the MR : 
https://gitlab.com/gitlab-com/migration/merge_requests/202

The repo is : git@gitlab.com:gitlab-com/migration.git

Create the steps for the maintenance for the Patroni failover.
Close/Open traffic.
Redirect applications for the correct Load balancers/ databases.

Please everything automated. Won't be allowed any direct interaction on hosts. All actions from the bastions.

Ahmad / Ongres are working on the postgresql.

@glopezfernandez @andrewn @dawsmith  @3manuek  @abel3  @ahmadsherif fyi

Story points: 2.0
Review and Clean-up PagerDuty Alerts

Check and review the source events that are generating alerts.

@dsylva  could you please add your report here?

Design Doc : https://docs.google.com/document/d/1dgpACz_clUNLdRRipxUqwlAAg3grG7A6wad0miHyezw

Number of incidents: https://docs.google.com/spreadsheets/d/1m3Qp-9fLCKSw38o3t48tlUbSwlW6F6Ymq-5Zxg8EPBw

Story points: 5.0
Stackdriver logs are missing HAProxy logs

Currently, stackdriver appears to be missing large volumes of haproxy logs.

Related slack thread: https://gitlab.slack.com/archives/CB3LSMEJV/p1542275675780800

This may be related to last week's haproxy slowdown: https://gitlab.com/gitlab-com/gl-infra/production/issues/553

Example: https://console.cloud.google.com/logs/viewer?authuser=0&project=gitlab-production&minLogLevel=0&expandAll=false&customFacets&limitCustomFacetWidth=true&dateRangeStart=2018-11-14T09%3A53%3A20.960Z&dateRangeEnd=2018-11-15T09%3A53%3A20.960Z&interval=P1D&resource=gce_instance&scrollTimestamp=2018-11-14T14%3A58%3A19.013464014Z&filters=text%3Ahaproxy&advancedFilter=resource.type%3D%22gce_instance%22%0Alabels.tag%3D%22haproxy%22%0AjsonPayload.frontend_name!%3D%22pages_https%22%0AjsonPayload.backend!%3D%22registry%22%0AjsonPayload.frontend_name!%3D%22pages_http%22%0AjsonPayload.frontend_name!%3D%22altssh%22%0AjsonPayload.environment%3D%22gprd%22%0AjsonPayload.server!%3D%22web-cny-01-sv-gprd%22%0AjsonPayload.server!%3D%22web-cny-02-sv-gprd%22%0AjsonPayload.server!%3D%22git-cny-01-sv-gprd%22%0AjsonPayload.server!%3D%22git-cny-02-sv-gprd%22

Story points: 2.0
Block search crawlers for all related domains

@plafoucriere discovered assets.gitlab-static.net URLs are indexed by Google. [original message](https://gitlab.slack.com/archives/C0259241C/p1542242346577000)

He also pointed out results for gprd.gitlab.com show up in Google as well.

[Results with assets.gitlab-static.net](https://www.google.com/search?source=hp&ei=4tjsW4qcLYXn_QaN_72wBA&q=site%3Aassets.gitlab-static.net&btnK=Google+Search&oq=site%3Aassets.gitlab-static.net&gs_l=psy-ab.3...3962.12004..12186...2.0..0.186.1815.30j2......0....1..gws-wiz.....0..0j0i131j0i3j0i10.LdbggfhUnlk)  
[Results with gprd.gitlab.com](https://www.google.com/search?ei=8djsW67yG867ggeegruQDg&q=site%3Agprd.gitlab.com&oq=site%3Agprd.gitlab.com&gs_l=psy-ab.3...34064.39612..39985...0.0..0.186.987.17j1......0....1..gws-wiz.rUoOGCRUsE4)

What are our best options to manage the crawl status for these domains and any subdomains?  I'm hoping we can make a simple robots.txt update to fix both of these. 😄

Do we have any documentation with domains we use across GitLab? I'd love to go through this list and find any other outlying index issues.

cc @sytses @lbanks

Story points: 1.0
Add nginx config test before restart nginx for about.gitlab.com

In https://gitlab.com/gitlab-com/gl-infra/production/issues/560, about.gitlab.com went down because of bad redirects. This can be immediately remedied by forcing a config test BEFORE restarting nginx. Currently the about.gitlab.com cookbook just restarts nginx if the config changes. 

- [cookbook-about-gitlab-com](https://gitlab.com/gitlab-cookbooks/cookbook-about-gitlab-com)
- [relevant recipe](https://gitlab.com/gitlab-cookbooks/cookbook-about-gitlab-com/blob/master/recipes/nginx.rb#L13-17)

Story points: 5.0
Admin access to staging.gitlab.com for Tony Carella

* Grant admin access to Tony Carella (acarella@gitlab.com) on staging.gitlab.com
* Supports the security team, as they are taking some responsibilities for IT Operations at this time
* No one on the security team has admin access to staging at this time. This will help us cater to access requests coming in for staging.
* access-requests


Story points: 1.0
Evaluate Ubuntu Advantage for livepatching support

Ubuntu offers the [Ubuntu Advantage programme](https://gitlab.slack.com/archives/CB3LSMEJV/p1542155941686600) (at a cost) which offers many benefits, a particular benefit for us is livepatching. With a large fleet of instances and major concerns about what can be rebooted, and if rebooted, in which order, live patching of the e.g. kernel becomes very relevant. Additionally, it could help mitigate attacks such as Spectre and Meltdown, which require urgent attention for everything, particularly public cloud hosted instances. The latest example of [speculative execution attacks](https://arstechnica.com/gadgets/2018/11/spectre-meltdown-researchers-unveil-7-more-speculative-execution-attacks/) just enforces the need for such focus and feature. 

The cost of Ubuntu Advantage may seem high ($250 per instance per year) but I believe the benefits outweigh the cost. Listed below are some of the benefits, I can think of:

* (Official Ubuntu benefit) Landscape, the Ubuntu server and desktop management tool
* (Official Ubuntu benefit) 24/7 telephone and online support portal
* (Official Ubuntu benefit) Canonical Livepatch Service
* (Official Ubuntu benefit) Ubuntu 12.04 Extended Security Maintenance
* (Personal opinion) Reduce cost by not having individual(s) tasked with preparing and executing reboots
* (Personal opinion) Reduce potential financial and reputational cost of failed reboots and failed reboot chains
* (Personal opinion) Reduce security risk and improve security posture by mitigating 0-day and similar attacks
* (Personal opinion) Provide overview of fleet and state of the operating system of the fleet

Disadvantages:
* Introducing a tool which makes automated changes to our environment which may impact service stability
* Cost 
* ... probably more I can't think of right now ...

I do not currently have access to data about the number of instances but to reduce cost, it could be considered to use Ubuntu Advantage for only a select few instances that are troublesome to reboot or instances that are public facing. My preference would be to cover the entire fleet but that may be financially unvailable. However, my impression is that GitLab infrastructure has a number of less used instances, which could be shuttered to reduce the overall cost of the infrastructure. Maybe this could even offset some of the Ubuntu Advantage cost.

Any thoughts on the above are very welcome.

Thanks,
Peter

Story points: 1.0
Mailroom locking/arbitration problem

Just raised https://github.com/tpitale/mail_room/issues/84 and noting it here for any local discussion we might like to have.

Noticed because we've currently got >30 messages in this state (of various ages up to 4-5 months).

I wonder if https://gitlab.com/gitlab-org/gitlab-ce/issues/2870#note_4184125 might be a way forward, but I'm not yet competent enough in Redis client use to comment well.

/cc @DouweM - you might be interested as the author of the original patch for mail_room.

Story points: 1.0
Windows machines for GitLab Runner CI tests

For GitLab Runner CI we need two more Windows machines:

- one with Windows 1803
- one with Windows 1809

On existing ones we're executing basic tests, building images for Windows Docker Containers and we will in near future start using `docker-windows` for running tests there.

The two new will be used to test support Docker for Windows with Linux containers. For this we need to ensure that HyperV is installed on the machines.

It will be enough if we will get the machines with proper Windows installed on them and with HyperV for Docker (some guidance can be found at https://blogs.msdn.microsoft.com/azuredev/2018/09/09/docker-in-azure-vm/). ~Verify can next handle installation and registration of Runner, installation of Docker and coupling it all together.

Story points: 1.0
Git Storage Node Repository metrics

I'd be interested in knowing the following about our Git storage nodes:

* number of repositories per node (absolute value)
* number of repositories per user (min, max, average, distribution) -- per node and across the fleet
* size of repositories (min, max, average, distribution) -- per node and across the fleet
* number of repositories based on activity (distribution: last hour, last 6 hours, last 12 hours, last 24 hours, last 3 days, last 7 days, last 2 weeks, last month, last 3 months, last 6 months, last 9 months, last 12 months, and everything else)

We should keep track of these metrics on an ongoing basis.

This request is intended to be able to help us guide decisions relevant to storage architecture.

Story points: 4.0
Add PathFactory to the CNAME for branded URL

We (Mktg) are bringing on a new tool that will handled content distribution and syndication. 
Can we please get a branded URL set up for this tool. 

[PathFactory Instructions](https://lookbookhq.force.com/nook/s/article/how-to-set-up-custom-subdomain)  

### Desired CNAME: learn.gitlab.com 

MktgOps main point of contact: @nlarue 

Please let me know if there are questions. 

/cc @nlarue @northrup

Story points: 2.0
Bump bootstrap script to version 8

The latest version of the bootstrap script (v8) includes a check to ensure that the requested kernel version is available before removing the current running kernel and attempting the upgrade (since it will naturally fail). However, we are not (yet) using that bootstrap script as the default everywhere, yet.

This issue will track the effort to update all version references for module source lines to ensure that version [v1.0.2](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/google/bootstrap/tree/v1.0.2) of the module is loaded and all environments are referencing `v8` as the default version of the script.

In order to prevent terraform from unnecessarily rebooting running instances, we will also have to use the `gcloud` CLI to update the `startup_script` metadata key on running instances.

Story points: 3.0
PackageCloud backups 'failing' according to DMS

Worked on May 4 and 13th, but haven't completed with exit code 0 since.

The error from the logs is
```ERROR: Upload of '/var/opt/packagecloud/backups/packagecloud-streamed-database-backup.1551991511.xbstream' part 11 failed. Aborting multipart upload.``` 
but the s3 bucket looks like it is correct, at least in list of files and byte sizes. 

Something weird is going on

Story points: 1.0
Break down postgres upgrade OKR

We have this OKR: `Upgrade Postgres to version 11 with log centralization and reporting.` captured in
* milestone: https://gitlab.com/groups/gitlab-com/gl-infra/-/milestones/22.
* epic: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/67

I propose to split this apart into multiple epics:

1. Postgres major upgrade
1. Log centralization
1. Reporting

Those are largely independent chunks of work we can parallelize or address independently across teams.

cc @ansdval

Story points: 1.0
Evaluate usage of more recent ZoL versions on -gcp kernels

ZoL 0.8.0 has been released, where as Ubuntu Bionic ships with 0.7.x.

[ZoL 0.8.0 has lots of improvements, that we want to use if possible.](https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0)

We will have to compile and package it our-own. This also gives us the ability to apply patches as needed. (Including patches that violate the GPL, as long as we do not redistribute those and the binaries, see https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLRequireSourcePostedPublic)

I have developed a prototype CI pipeline to compile the kernel modules here: https://gitlab.com/gitlab-com/gl-infra/gitlab-zfs/

Tl;Dr; of the comments: It works. Bionic & xenial. But we'll not be running it in prod (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6883#note_180038845)

Story points: 4.0
Upgrade to PgBouncer 1.12.0

Pgbouncer 1.12.0 has been released.


We should look into upgrading PgBouncer accordingly.

Description of the change : https://gitlab.com/gitlab-com/gl-infra/production/issues/1325

This maintenance would be done to a critical component from the platform so we should schedule to a low peak time as a c1.

For the read only nodes:

* Stop the traffic for the node, and remove the node that could become a primary database.
* execute a chef cookbook to uninstall and install the newer version of pgbouncer (or upgrade), keep the same configurations. -Restore the traffic and make the node again available to become a primary database.

For the pgbouncer nodes ( that we will have 2 fleets with 3 nodes), execute in all the nodes :

* take a node out from the rotation on the ILB.
* Update the config from the other 2 nodes from pgbouncer to add temporarily the traffic from the node that is out.
* execute a chef cookbook to uninstall and install the newer version of pgbouncer (or upgrade), keep the same configurations.
* restore the config from the 2 nodes from pgbouncer.
* Add the node back to the rotation on the ILB.

Story points: 20.0
Codify update rules for GitLab Status page

Our customers&mdash;internal and external&mdash;rely on status.gitlab.com to determine whether an issue they're experiencing is known by our support and reliability teams. The status page is also the primary means for broadcasting public information about incidents. To ensure this is always the case, we will produce a set of rules in the on-call checklist. We should update our tools to communicate around the status updates, thus ensuring the status page is not a secondary medium for communication.

Story points: 2.0
SREs should all have Zendesk "Light Agent" access

As outlined in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/23124, all SREs when they're onboarded should setup Zendesk "Light Agent" account access. Because this addition to the list was added after everyone's start date, please follow the instructions provided by [internal-support](https://about.gitlab.com/handbook/support/internal-support/#light-agent-zendesk-accounts-available-for-all-gitlab-staff) to setup your access.

Please confirm:

- [x] @abrandl 
- [x] @ahanselka 
- [x] @ahmadsherif 
- [x] @alejandro 
- [x] @cmcfarland 
- [x] @craig 
- [x] @craigf 
- [x] @cmiskell 
- [x] @cshobe 
- [x] @dawsmith 
- [x] @devin 
- [x] @Finotto 
- [ ] @glopezfernandez 
- [x] @hphilipps 
- [x] @mwasilewski-gitlab 
- [x] @msmiley 
- [x] @nnelson 
- [x] @yguo 

Optional:

- [ ] @andrewn 
- [ ] @jarv 
- [ ] @marin 
- [x] @skarbek

Story points: 1.0
improve incident management tooling

There are several open issues to improve incident management which need to get finished:

- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5543
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6424
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5359
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5508 (was this ever used?)
- https://gitlab.com/gitlab-com/runbooks/issues/19

We also should try to reduce the number of bots that need to be used and make them easy and consistent to use and test.

During last incident the `/start-incident` slack command failed to create an incident issue. We should regularly test our tooling and think about regular incident trainings.

Story points: 0.0
consolidate incident management documentation

We have several issues with our incident management documentation:

- we have documentation in several places with different/outdated content:
  - https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/
  - https://gitlab.com/gitlab-com/runbooks/blob/master/howto/manage-production-incidents.md
  - https://gitlab.com/gitlab-com/runbooks/blob/master/incidents/general_incidents.md
  - https://gitlab.com/gitlab-com/runbooks/blob/master/incidents/database.md
- The handbook page is a very long read which is fine to learn all about the processes if you have time but does not work if you have an incident and need to figure what to do ASAP.
- We have no documentation on
  - how to find the EOC/IMOC/CMOC
  - how to page the EOC/IMOC/CMOC
  - chatbots and their usage

We need to consolidate this documentation and make it easy to find and useful for everybody during an incident.

Story points: 5.0
Upgrade to terraform 0.12

Terraform 0.12 [has been officially released](https://www.hashicorp.com/blog/announcing-terraform-0-12), and contains many improvements based on the corresponding updates to HCL that we can leverage in our infrastructure code. This issue is to track efforts for working through the [upgrade guide](https://www.terraform.io/upgrade-guides/0-12.html) on all of our environments in gitlab-com/gitlab-com-infrastructure>

* [x] update all envs in gitlab-com-infrastructure to 0.11.14
* [x] update terraform modules to 0.11.14
* [x] update all environments to use new versions of modules
* [x] update terraform modules to 0.12
* [x] update all environments to 0.12, I tested it on gstg and it was clean, @craigf already did some work here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7067

/cc @ansdval @dawsmith @Finotto FYI for prioritization

Story points: 13.0
Complete the container registry deployment for a kubernetes cluster in pre

We've built a [PoC] in the past, let's move from PoC to a [real environment].

This is the realization of decisions made in this issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5337

[PoC]: https://gitlab.com/gitlab-com/gl-infra/kubernetes-poc
[real environment]: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com

Story points: 5.0
Update the terraform gke module to provide the ability to separate how node pools are created

Our current GKE module relies on the use of the default node pool.  This is a great default, but limits our ability to customize a cluster in the future in the cases where we need beefier nodes for X reason, or if we need to enable or disable preemptible instances for some other reason.

Disable the default node pool and provides the ability to configure many node pools.

Story points: 3.0
Update the terraform gke module with the ability to lock down what networks have access to the kube api server

By default GKE clusters have open access from all of the Internet to access the API server.  We should limit this, or at LEAST provide a mechanism to update this list in the future.

https://cloud.google.com/kubernetes-engine/docs/how-to/authorized-networks

https://www.terraform.io/docs/providers/google/r/container_cluster.html#master_authorized_networks_config

Story points: 1.0
Update the terraform gke module to provide the option to enable network policies feature of GKE

[GKE Network Policy](https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#restrict_with_network_policy) provides a method of locking down traffic inside of the Kubernetes cluster.  Update our module to provide a method of enabling this optionally with some form of configuration.

https://www.terraform.io/docs/providers/google/r/container_cluster.html#network_policy

Story points: 1.0
Update the terraform gke module with the ability to create private clusters

In general, there really isn't a reason why our clusters are public.  These nodes all have public ip addresses and all egress traffic comes from whatever that nodes IP address is.  Let's fix this.  This will improve our security stature by limiting what IP addresses we expose to the outside world, this will make it easier for the community as we'll have a set IP address where our traffic comes from, and it just feels like a cleaner more sane configuration.

This will require the use of a NAT of some sort, otherwise egress traffic won't know how to reach the Internet.

Story points: 3.0