giseldo/neodataset huggingface dataset:
GKE in GPRD is evicting our registry Pods

Occasional checks into the cluster as traffic in canary has shifted has resulted in some evicted Pods over the course of time.  We do not yet have alerts set for this type of situation.  To be handled here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7604

Utilize this issue to track what we can do to prevent Evicted Pods.  The reasoning has currently been diagnosed as memory overuse compared to the configured memory requests:

```
Message:            The node was low on resource: memory. Container registry was using 3309728Ki, which exceeds its request of 32Mi.
```

This particular Pod in the above example was using over ~413MB of RAM, but we only request ~32MB.  The requests configuration:

```yaml
   registry:
    Image:      registry:2.7.1
    Port:       <none>
    Host Port:  <none>
    Requests:
      cpu:        50m
      memory:     32Mi
```

Utilize this issue to track how best to handle this situation.  It may be that we need to figure out an appropriate baseline and manage our requests appropriately such that Kubernetes will scale the nodes as needed.

Reference: 
* https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
* https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/

/cc @gitlab-org/delivery

Story points: 3.0
ops.gitlab.net slow-down due to the database and VM being in different regions

the WIP issue https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6675 has left the ops in a state where it is very slow. we are also seeing some timeout errors from chatops:

![Screen_Shot_2019-08-20_at_4.54.33_PM](/uploads/32172e4bf04d337df1c501b8b8dec7e0/Screen_Shot_2019-08-20_at_4.54.33_PM.png)


To wrap this up we can do the following:

1. Delete ops-too
2. Create a new instance with a new name
3. Mount nfs, setup to use memorystore, connect to cloudsql (without repos at this point)
4. Shut down both boxes, rsync data from the old to the new server
5. change the dns

Or if we are worried about introducing too many changes, keep files on local disk.

Story points: 2.0
Run ClearSharedRunnersMinutesWorker on production

In https://gitlab.com/gitlab-org/gitlab-ce/issues/65540 we fixed an issue where the pipeline minutes usage was showing incorrect values.
The problem was in [`ClearSharedRunnersMinutesWorker`](https://gitlab.com/gitlab-org/gitlab-ee/blob/0c1c17abba98ffabbb59e854672cef60c8803e39/ee/app/workers/clear_shared_runners_minutes_worker.rb) that was timing out when trying to reset numbers for projects.

Given that the fix is out now, we would need to run it on production to reset the pipeline minutes quota so that users can see correct statistics. [We have decided to do so](https://gitlab.com/gitlab-org/gitlab-ce/issues/65540) now despite giving free minutes to users.

The same worker should run automatically on cron schedule on the 1st of every month.

### The ask

is it possible to run the following on production?

```ruby
ClearSharedRunnersMinutesWorker.new.perform
```

and report the status of the operation in this issue. This should not take long and the result would be that project and namespace statistics will be in sync for the month of August.

/cc @drewcimino @ahanselka

Story points: 1.0
Route SLO alerts to pagerduty

https://gitlab.com/gitlab-com/runbooks/merge_requests/1344 is routing latency SLO alerts to pagerduty as they seem to be very accurate in indicating real production issues.

We also should send other SLO alerts in https://gitlab.com/gitlab-com/runbooks/blob/master/rules/general-service-alerts.yml to pagerduty after we trimmed down the false alert rate (https://gitlab.com/gitlab-org/gitlab-ce/issues/66166):

* [ ] error ratios alerts
* [ ] saturation alerts
* [ ] service availability alerts

Operation rate alerts probably shouldn't go to pagerduty, as they will manifest in higher latencies or error ratios when they affect production.

Story points: 3.0
Shut down cron.gitlab.com

This machine was only used for the Zoom sync (https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1659), and this has now been replaced with a schedule pipeline (https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/28319).

- [x] chef-repo changes: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2174
- [x] gitlab-com-infrastructure changes: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/1191

/cc: @ahanselka

Story points: 2.0
Bolster Alerting for GKE clusters and components

We are lacking in alerting/paging for issues with our GKE Clusters and components installed inside of them.  Let's see what we can do to improve our stature on this.

Consider the following:
* [x] The `stable/prometheus-operator` helm chart has many alerts that we've chosen to not implement due to our own configuration for alerting.  Consider porting these over if they seem to suit our needs
* [x] Alert for any number of evicted Pods that remain around - https://gitlab.com/gitlab-com/runbooks/merge_requests/1357
* [x] Alert if we've been unable to scrape metrics for an extended period of time `kube_hpa_status_condition` https://gitlab.com/gitlab-com/runbooks/merge_requests/1376
* [x] Custom rules for `gitlab_component_availability` need to capture our running Pods - https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7575
* [x] Validate rules are in place for when we've reached the limits of our HPA scaling configurations https://gitlab.com/gitlab-com/runbooks/merge_requests/1376
* [x] can we alert when we are bumping against the maximum of allowed nodes in a node pool? https://gitlab.com/gitlab-com/runbooks/merge_requests/1377
* [x] node specific metrics, disk, memory, cpu usage
* [x] container throttling - https://gitlab.com/gitlab-com/runbooks/merge_requests/1407

Story points: 5.0
Consolidate bastion `howto` pages in runbooks

We currently have the following in gitlab-com/runbooks:

```
howto/gprd-bastions.md
howto/gstg-bastions.md
howto/ops-bastions.md
howto/dr-bastions.md
howto/pre-bastions.md
```

At a minimum, this should be consolidate. I prefer we instead make this a file in this repository's onboarding directory. We should update it there regularly.

Story points: 1.0
When a deployment to Kubernetes fails, the master branch is now inconsistent with what has been deployed

Our deployment procedure for Kubernetes assumes that deployments would always succeed.  In times for which they do not, helm will automatically roll back the desired change.  This presents an issue as the master branch is no longer an accurrate representation for what is running on the cluster.  Utilize this issue to track what we can do to ensure that master is a reflection of what exists on a cluster.

## Thoughts

* We would want to setup a failure job that performs work, potentially reverting the change
* Issues with linked MR's would need some form of communication to know that a deploy was unsuccessful, anything that may have been auto closed may need to be reopened

/cc @gitlab-org/delivery

Story points: 1.0
GPRD Kubernetes Cluster is generating many errors

Utilize this issue to track these errors and learn/discuss/address.

https://log.gitlab.net/goto/5770bdf4c1f9aa4707de3a60fcd6e37b#/?_g=h@44136fa

| Log | Filter? | Ignorable | Info |
| --- | ------- | --------- | ---- |
| `daemonsets/fluentd-gcp-v3.2.0 was not changed` | possible | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204637830 |
| `rm: can't remove '/etc/ssl/certs/*': No such file or directory` | possible | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204643312 |
| `WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping` | possible | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204643312 |
| `unable to fetch pod metrics for pod <NAMESPACE>/<PODNAME>: no metrics known for pod` | Not advised | No | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204658792 | 
| `...watch of \*v1.Endpoints ended with: too old resource version...` | possible | Possibly | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204660050 |
| `caller=shipper.go:350 msg="upload new block"...` | Not advised | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204660110 |
| prometheus info level messages | No advised | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_204813763 |
| `time="<TIMESTAMP>" level=info msg="debug server listening :5001"` | Not advised | Yes | https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_205157205 |

### Actionable Items

https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7577#note_205227095

Story points: 3.0
Investigate saturation of read-only replica PGbouncer (and lack of load distribution)

During https://gitlab.com/gitlab-com/gl-infra/production/issues/1054, we observed the following behavior:

* At 8:30a UTC `patroni01` experienced a network issue that caused Patroni to fail over to `patroni04`
  * `patroni01` ended up in a corrupted state and was offline
  * `patroni04` became the new master
  * `1` master, `5` replicas

* `patroni06` immediately became overloaded: 
  * server connections maxed out at 100 (as configured)
  * active clients dropped
  * waiting connections increased

![Screen_Shot_2019-08-14_at_9.22.28_PM](/uploads/99de8c23fc6a81c02ae14e4c23b9b29a/Screen_Shot_2019-08-14_at_9.22.28_PM.png)

* however, all other replicas, while registering the failover blip, managed to stabilize nearly inmediately:

![Screen_Shot_2019-08-14_at_9.27.34_PM](/uploads/af8e0f6aef112449bb205b431b4f225b/Screen_Shot_2019-08-14_at_9.27.34_PM.png)

The behavior of `06` wasn't expected. As an experiment, we took `06` out of the cluster temporarily. What we observed was `03` crater under the excess load:

![Screen_Shot_2019-08-15_at_12.10.40_AM](/uploads/454b1843f3d5de33ffd2ac6a484cbd55/Screen_Shot_2019-08-15_at_12.10.40_AM.png)

When `01` was finally restored an added to the cluster as a read-only replica, we attemped the experiement of pulling 06 out of the rotor. As 03 did, it cratered:

![Screen_Shot_2019-08-15_at_12.12.25_AM](/uploads/9cd6b5a9f969ebbbbb6444ffbbd7d7dd/Screen_Shot_2019-08-15_at_12.12.25_AM.png)

@stanhu checked the internal list of hosts (no screenshot or data saved) and it simply did not seem to have enough entropy. From memory, `05` was the first database replica in the list. After we added `01` to the cluster, `05` was still first in the list and `01` was last. With a sample of 1 observation, the combination we saw post rejoin is possible, but it seems unlikely. We expected not to see `05` in the same spot as before, and we didn't expect to see `01` last.

It is worth noting that once we were back to 5 replicas, we started seeing a slow recovery on `06`:

![Screen_Shot_2019-08-15_at_12.23.27_AM](/uploads/6584dfc0630308f546742333691422e6/Screen_Shot_2019-08-15_at_12.23.27_AM.png)

Thus, there is clearly a capacity component to this riddle. As a precaution, we added another database replica, `07`, in hopes that this addition will buy us some runway in case of having another replica fail.

Story points: 3.0
Client-side read-replica list observability

During https://gitlab.com/gitlab-com/gl-infra/production/issues/1054, we observed unexpected behavior from the application read-replica load balancing. When the `01` node failed over to `04`, the failover caused significant, uneven load to shift to `06`. When we removed `06` from the cluster, the load shifted to `03`. When we undid that, the load came back to `03`. When we added the restored `01` and tried to remove `06`, `01` became overloaded. It was only when we had all replicas working and let them have time to stabilize that we saw the platform go back to nominal operation.

One of the questions we asked during the incident was "what does the list look like across the fleet". It turns out we can't easily answer that. @stanhu did some magic and was able to watch this on a few nodes:

![Screen_Shot_2019-08-14_at_3.15.21_PM](/uploads/e02f5543900cac729f2acb5b3445b74a/Screen_Shot_2019-08-14_at_3.15.21_PM.png)

It would be useful to expose this data and be able to see it site-wide. Is it really as randomized as we think it should be? Exposing this in logs might be a first step, but it would required on-the-flight processing. A possible visual approach is, for each replica, how many clients have it in each position in the array. We may cap that at 4 positions to start this, regardless of the number of replicas). In any event, being able to see this during an incident likes today's is imperative.

Story points: 3.0
Patroni documentation and training

Post https://gitlab.com/gitlab-com/gl-infra/production/issues/1054, we realized that ee need to do a review of Patroni documentation and runbooks and do training on how the database cluster is configured, how Patroni works, how we perform common operations (add a node, remove a node, check on status, build a node, etc).

Everyone on-call (EOCs and IMOCs) must become familiar with this. DNA seems like a good setting to run such training.

Story points: 3.0
Patroni runbooks and tooling => 20%

During https://gitlab.com/gitlab-com/gl-infra/production/issues/1054, it was apparent some of our Patroni runbooks lacked details, didn't work, or were out of date. For instance, creating a new replica ran into a bunch of issues, and there were questions about the validity of the Chef/Terraform state: disk sizing, Patroni startup, etc.

Also, this still requires deep knowledge of flags, and such. We should think about abstracting some of these operations in a tool that hides this complexity. I want to be able to type something like 'foo create replica --production' or 'foo disable replica --production <target>', where *foo* is likely `gittab-ctl patroni`.

The way we do this today is too risky, as we craft commands on the fly and execute them on hosts in production.

Story points: 3.0
Build automation for calculation and storage of MTBF for user facing services

Sub issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7431.

Starting point to discuss the question on how we track MTBF.

Currently we have a [spreadsheet here](https://docs.google.com/spreadsheets/d/1bCEUQbWMccVm1dfBrOFtXjyDVCflHu4rB6htOaC_0yI/edit#gid=1150439504), but it is manual..

Questions for discussion here:
1.  Where are we going to get the data?  Currently production issues.
1.  What are the MTBF metrics we want to track? 
  - MTBF between Sev 1/2 incident issues
  - MTBF between any incident issue
  - Further breakdown by tags - eg MTBF by service ?
1.  Where do we want to store the end result - prometheus, periscope?

adding @alejandro @cmcfarland @devin @ahanselka @nnelson @ggillies to start discussion on these

Story points: 1.0
Cert-manager versions older than 0.8.0 need to be upgraded by Nov 1st

Letsencrypt says:
```
We've been working with Jetstack, the authors of cert-manager, on a 
series of fixes to the client. Cert-manager sometimes falls into a 
traffic pattern where it sends really excessive traffic to Let's 
Encrypt's servers, continuously. To mitigate this, we plan to start 
blocking all traffic from cert-manager versions less than 0.8.0 (the 
current semver minor release), as of November 1, 2019. Please upgrade 
all of your cert-manager instances before then.
```

Some of our older Kubernetes clusters fall into this category - probably because of autodevops setups which run cert-manager pods.

/cc @skarbek

Story points: 2.0
Deep Dive on production#1039

@aamarsanaa @hphilipps i please drive a deep dive on the incident https://gitlab.com/gitlab-com/gl-infra/production/issues/1039. we'll perform a walkthrough of the incident during the DNA meeting on 14 August. Please re-familiarize yourselves with the incident and RCA. This issue has much more to do with process than troubleshooting.

Story points: 1.0
Replace SSL cert for dashboards.gitlab.com

``` 
https://dashboards.gitlab.com
 SSL certificate for https://dashboards.gitlab.com expires in 4d 23h 24m 58s
```
-----
```
$ echo "" |openssl s_client -connect dashboards.gitlab.com:443 -showcerts 2>&1|openssl x509 -noout -enddate
notAfter=Aug 16 23:59:59 2019 GMT
```
-----
```
sslmate list |grep dashboards.gitlab.com
dashboards.gitlab.com                                   DV    Active   2020-08-17  No key file 
```
-----
Certificates stored/managed in gcloud, so we'll have to upload an entire new one, which means we need to rekey in sslmate.

Story points: 1.0
Clean up of Production Board

The Production Board (https://gitlab.com/gitlab-com/gl-infra/production/-/boards/1204483) has a number of items in the `Open` state but do not seem like incidents, changes or deltas. We need to clean those up, either by moving them to Infrastructure project or closing them out.

Additionally, there are a bunch of changes which aren't really changes or were changes at some point which were unfinished. As a general rule, a change has clearly defined start and end states. Let's clean that up as well, either by finishing them or moving them to be Infra issues.

Story points: 2.0
sshguard Blocking access to consul servers

`sshguard` is blocking ssh access to consul servers.  It is getting requests from the bastions which it interprets as hostile and adds an iptables rule to block the bastions on the ssh port.  This has the effect of locking everyone out. We should either set a timeout that allows traffic again, or use a different method of determining hostile traffic. 

```
sudo iptables -nvL --line
```
Results in lines like the folowing - sometimes 2 or 3 bastion servers are listed

```
Chain sshguard (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1    14214  935K DROP       all  --  *      *       10.216.4.4           0.0.0.0/0
```

The line can be removed with

```
sudo iptables -D sshguard 1
```

We should also determine exactly which traffic is triggering this behavior.  If it is not being caused by someone with access who has their ssh setup misconfigured, then it is being caused by someone malicious who has gotten access to the bastions.

Story points: 1.0
Work out exactly how many merge requests were closed in https://gitlab.com/gitlab-com/gl-infra/production/issues/1040

In https://gitlab.com/gitlab-com/gl-infra/production/issues/1040 a race-condition between the PostReceiveWorker and the Merge Request Refresh Worker (?) related to delayed cache invalidation led to merge requests getting closed by a bug in the application.

At present we do not know how many were closed.

The decision has been made not to automatically reopen the MRs as this may lead to further confusion for users, however it's important from a messaging point-of-view that we understand roughly how many users were affected by this bug.

@abrandl has done some initial analysis so assigning to him. 

cc @smcgivern for any insight into how we can determine whether an MR was closed by a missing branch or a user action.

Story points: 0.0
Terraform GKE Module failure to apply network policy

When the network policy is disabled as desired, we still attempt to configure the provider, which fails:

```
1 error occurred:
        * module.gke-skarbek.google_container_cluster.cluster: 1 error occurred:
        * google_container_cluster.cluster: googleapi: Error 400: The network policy addon must be enabled before updating the nodes., badRequest
```

Story points: 3.0
Terraform GKE module unable to upgrade Kubernetes version

When attempting to upgrade the cluster using terraform, we fail:

```
1 error occurred:
        * module.gitlab-gke.google_container_cluster.cluster: 1 error occurred:
        * google_container_cluster.cluster: node_version was updated but default-pool was not found. To update the version for a non-default pool, use the version attribute on that pool.
```

Story points: 3.0
Create Production GKE Clusters

Utilize this issue to:
* discuss how we want to spec our production GKE cluster(s)
* decide if we should have 1 cluster for `production` and cluster for `cny`, or if 1 cluster can handle both
* After decisions are made, either spin up a new issue, or reuse this one to complete the work

Our design documents did not go into the desired detail to complete this implementation.

/cc @gitlab-org/delivery 
/cc @gitlab-com/gl-infra

Story points: 3.0
Improve error-handling in Postgres graceful-failover script

**Goals:**

Improve the error-handling of our [`graceful-failover` script](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/bin/graceful-failover):
* The script appears to have killed itself during error-handling, before calling its `resume_pgbouncer` function.
* The deadlines of 10 and 5 seconds appears to not always be long enough to complete all of the PgBouncer PAUSE operations.  Consider tuning those deadlines.

**Background:**

The [`graceful-failover` script](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/bin/graceful-failover) automates the steps of switching which postgres instance is currently the primary (writable) node:
* Identify the current primary member of the given environment's Patroni cluster.
* Issue the `PAUSE` command to the PgBouncer instances used by db clients that need write access.
* Wait for up to 10 seconds for the `PAUSE` to complete on all PgBouncer instances.
* If still waiting, kill all active or idle-in-transaction db sessions on the primary Postgres node.
* Wait for up to 5 more seconds for the `PAUSE` to complete on all PgBouncer instances.
* If still waiting, try to abort: Kill the `knife` command that was trying to issue the `PAUSE` commands, issue a `RESUME` command to all PgBouncers, and then exit the script.
* At this point in the script, assume the `PAUSE` succeeded.
* Tell Patroni to run the switchover immediately, preferring the specified node as the new primary.
* Unpause all PgBouncer instances.

During today's production maintenance, the script had 2 problems:
1. The PAUSE command was slow enough to reach the script's 1st and 2nd deadline.  That is abnormal, and we should look for ways to reduce the chances of that happening.  Note that the PgBouncer [PAUSE command](https://pgbouncer.github.io/usage.html#pause-db) is a blocking operation, so the duration is partially dependent on client behavior.
2. The error-handling routine failed to reach the line that would un-pause the PgBouncer instances.  That left PgBouncers in a paused state, causing extended downtime.

Details of today's automation failure are logged in [this production change issue](https://gitlab.com/gitlab-com/gl-infra/production/issues/952#note_200807981).

Story points: 3.0
Automate terraform for env-projects

Update gitlab-ci.yml in gitlab-com/gitlab-com-infrastructure> to automatically plan/apply configurations in `environments/env-projects` as well as the other application-centric environments. This environment may need to be handled a little differently because of the inherent circular dependencies. There are multiple approaches to handle bootstrapping the remote state for the genesis project the first time, ([here](https://www.monterail.com/blog/chicken-or-egg-terraforms-remote-backend) is an example [with sample code](https://github.com/monterail/terraform-bootstrap-example))

Story points: 2.0