giseldo/neodataset huggingface dataset:
file-35-stor-gprd rebooted

The `file-35-stor-gprd.c.gitlab-production.internal` server rebooted at 03:39 UTC.

The following repositories have 0 byte files which may need to be deleted before pushes will succeed.
```
/2d/a6/2da627fc34bd145119f7c4b28180de9b1b1ecb04cd50395d785d1c69206b35b7.git
/ee/0e/ee0e418f314258adb918e4c885f7e69d24393d6c089416490541195ccf9ec8b9.git
/ee/98/ee98bb4e4bccdb9985892d934e1019d8a83c8106fff666d12b61fa48b61f923f.git
/88/02/8802266a8fdb5df071ff72872f5026a6cac54123e19916215c28faa6cb9c48c3.git
/a5/96/a596fce6a7c5a270336cf6e4d93a2e68f0f48cf10f9dfe49fb9091b75eb8477b.git
/f4/46/f446c5ffe314c2cf078dfa09e27fdec032fadba351ab17c39132699db0e56821.git
/71/a3/71a39ee9d13571f24e71db9481c6c79fe16ae18d46ae3ea0f702a7f2ab81b9fd.git
/03/b0/03b051a9df00021abd386e862448aae39f02cac86d6705d757868abf0ae181ff.git
/70/a8/70a8b032d6fb8a191a4262bea0b7ad1edf89069c9916c0fba205df356dbd9147.git
/db/e7/dbe7ad164b13687f1f4cb406c010c3baa1cdef40e3bed488dc78bd6253172ad9.git
/86/9d/869dfce8970926048a3fc4af2dc1f6a90a777b655f9d98588a38770e60c9a6e7.git
/a9/d6/a9d6b8b752a339be792c71051237e96d792e64a4654077d79c58a64e641a60b6.git
/50/97/5097eec0504007ccff80e78fccecf3f7967688c86f6e7b2efacf5b003acc7159.git
/69/42/6942e0cdf2a521808561da1b5e0b5255404c14b56551d3c69e2e5b6523791990.git
/cf/ea/cfeae16b595d765939a14e1406ab2068a4f03c1cb835a12c3b09e4b93319d758.git
/4b/f2/4bf2aeadb0603ae6c62c755864d5cde291a48ff7ac759b529105aa498ea60c92.git
/72/f2/72f2fde2baebc34149ad159e3d24051bb70685e8512a48163b3476c4d370d185.git
/6d/be/6dbee4cf8657d93310ee25693acce504b9972c2fa162ac41c9d056e12efcb8db.git
/30/6b/306b9095b0213e79aca371b230061b110a54186e2ac80336f8b8925725702e9d.git
/9e/de/9ede26caa0c4023e99c905ffec40ce1d5aad20f69bce64fde781d51a7e2d4c30.git
/49/65/4965abd0081151b6f547f562e730f763cff5417d2352380be4c0bd64dbddfcb1.git
/49/87/498797e5e80b416ee9471d61df9782116fb5917280982006cb543da8f7a6d4f2.git
```

These files are located in `/var/opt/gitlab/git-data/repositories/@hashed`

Story points: 2.0
Add new sitespeed graphite datasource to grafana

To finalise a completely new setup for doing live monitoring of overall frontend performance through sitespeed I would need the setup of a new graphite datasource in our grafana instance. This would then finalise the issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/2106

I have added the credentials + server address to 1P under `Sitespeed Graphite New`. It would be great to have this as a new datasource `sitespeed new`.

As a standard grafana member I can add new dashboards without problems or? (So far I tested them only on my local grafana instance)

Story points: 1.0
Decrease statement_timeout in staging

We have `statement_timeout` settings in gstg that are not consistent with production. We should align that, for example to allow issues with migrations to surface earlier before they're tried in production.

While the global `statemnt_timeout` in gstg is 15s, individual per-user settings are different:

```
root@patroni-06-db-gstg.c.gitlab-staging-1.internal:/var/opt/gitlab/postgresql# gitlab-psql -U gitlab
Password for user gitlab: 
psql (9.6.15)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

gitlabhq_production=> show statement_timeout;
 statement_timeout 
-------------------
 5min
(1 row)
```

The `statement_timeout` has been set individually for `gitlab` user, see `pg_user` table:

```
 gitlab            |    16384 | f           | f        | f       | f            | ******** |          | {statement_timeout=5min}
```

We should reset this to 15s to align with production. Since staging is a lot smaller than production, we might still not catch all issues - but that's another topic.

Story points: 1.0
Add detailed network monitoring for PostgreSQL/Patroni hosts

As I already proposed before in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7543#note_204263742 :

> We might want to consider having every node send/receive a continuous stream of low bandwidth traffic, and monitor it. Something like a permanently running `iptraf`. This cannot be 1m resolution time, as some already available metrics on Prometheus. With this, we might be able to see clearly if there were network hiccups.

I would like to raise the topic again. Prometheus network monitoring is good except for two data points that are not available:
*  Low resolution (1 second resolution is desired).
*  Dependent on the traffic.

I propose to setup a system to inject traffic between the hosts at a constant rate, and export metrics to Prometheus. This way, we can see what is the performance of the network vs the expected one. It may consist of the following:

*  Each host "monitors/talks to" two other hosts of the cluster, ideally one in the same zone and another one in another zone, in the same region.
*  A small amount of traffic, but constant, is exchanged (think of something like `dd if=/dev/zero | pv --rate-limit | nc` etc). For example, 50KB/s.
*  Export network performance metrics to Prometheus, ideally with a 1 second resolution.

This mechanism would allow to detect and measure precisely network hiccups and disruptions. It would help clearly to diagnose the recent Patroni failovers, which are believed to be caused by coordinated network disruptions (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7790#note_213872680).

cc @glopezfernandez

Story points: 2.0
Connect GitLab services to project environment variables

The environments built by [GitLab Services](https://ops.gitlab.net/gitlab-com/services-base) need the ability to associate environment variables with projects and groups within the GitLab application.

Some service environments will not require variables to connect (master, review apps).  Some will need to automatically set variables in an arbitrary project or group on GitLab.com.  Some will need to set variables in another instance that is not GitLab.com (staging, ops, dev instances).

The location of the project to connect to should be configurable using a Terraform variable in the files in the [/environments](https://ops.gitlab.net/gitlab-com/services-base/tree/master/environments) folder in the GitLab Services project. The absence of this variable should result in no attempt to set variables in any project or group.

Initially the two variables that there is initial need for are:

- [DATABASE_URL](https://docs.gitlab.com/ee/topics/autodevops/#postgresql-database-support) set from the [CloudSQL Module](https://ops.gitlab.net/gitlab-com/services-base/blob/master/terraform/main.tf#L75)
- [POSTGRES_ENABLED](https://docs.gitlab.com/ee/topics/autodevops/#database) and possibly other database variables.

When setting these variables, the additional attributes also need to be configurable

- Scope - to specify which environments the variable applies to
- Masked - to hide protected variables
- State - for completeness

Story points: 2.0
Connect GitLab services GKE clusters to GitLab Projects

The environments built by [GitLab Services](https://ops.gitlab.net/gitlab-com/services-base) need the ability to associate their GKE clusters with projects and groups within the GitLab application.

Some service environments will not have an automatic Kubernetes integration connection (master, review apps).  Some will need to automatically connect to an arbitrary project or group on GitLab.com.  Some will need to connect to another instance that is not GitLab.com (staging, ops, dev instances).

The location of the project to connect to should be configurable using a Terraform variable in the files in the [/environments](https://ops.gitlab.net/gitlab-com/services-base/tree/master/environments) folder in the GitLab Services project. The absence of this variable should result in no attempt to connect GKE to a project or group.

The three variables which need to be connected are:

- The CA Cert from the GKE cluster defined using the [GKE Module](https://ops.gitlab.net/gitlab-com/services-base/blob/master/terraform/main.tf#L39)
- The Endpoint IP of the GKE cluster defined above
- The Service token of the `gitlab-admin` account defined in the [gitlab-admin-service-account](https://ops.gitlab.net/gitlab-com/services-base/blob/master/kubernetes/gitlab-admin-service-account.yaml)

Additional settings are:

- `Environment Scope` - which should come from a configuration variable in the environment file and should default to `*`
- `Base Domain` - which can come from a configuration variable, and will eventually be sync'd with a DNS module
- `Cluster Name` - should default to the name of the branch or the `var.project` TF variable

This work should be done in an ephemeral environment and should be functional before it is merged with `master`.  Assign the MR to @devin for review.

Story points: 3.0
Stop using numbers in our host naming convention

Lately, I've observed considerable amounts of thought and attention given to replacing hosts and naming them to re-sequence the pool counting up from `01`. This pattern _always_ requires destruction or renaming, if even possible, of the host being replaced before the new host can be created. The extra amount of work this creates and the thought and attention given to such operations distracts us from the goal of destroying and replacing the broken VM.

:cow: :cow: :cow: :cow: :cow:


Cc @craig - I'm not going to assign this to you, but I know you'd come across a way of doing this with randomization and pools of silly names.

Story points: 3.0
Create similar alertmanager configurations across all instances of running alertmanagers

Today the alertmanager consists of three files:
* https://gitlab.com/gitlab-cookbooks/gitlab-alertmanager/blob/master/templates/default/alertmanager.yml.erb
* https://gitlab.com/gitlab-cookbooks/gitlab-alertmanager/tree/master/files/default/alertmanager/templates

The `alertmanager.yml.erb` is put together by chef and contains secret data that we store inside of GKMS.  We need to determine a way to recreate this workflow such that we can make the same modification to one set of files and these are then pulled in by both chef, and our desired mechanism for deploying the alertmanagers inside of Kubernetes.

Reference Doc: https://docs.google.com/document/d/1oJcWY4U2lmPwBvY8qQPlP7msPc7ioVUlo3ZKkCp-i1g/edit

Related Issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6927

/cc @gitlab-com/gl-infra

Story points: 5.0
Thanos reconfiguration

At the time of writing up this issue, the stable/prometheus-operator helm chart does not support the same version of thanos that we are running throughout the rest of our infrastructure.  The use of thanos is preferred to ensure the longevity of data beyond the time for which prometheus is able to store data in our GKE clusters.  Utilize this issue to discuss/investigate/create actionable issues to ensure that we get thanos enabled with the desired configuration in GKE, and that prometheus is configured in a preferred manner.

* [x] Check-up on which version the latest version of the stable/promethues-operator chart supports such that we can hopefully get the same version of thanos deployed across both our VM and Kubernetes infrastructure
* [x] Create a separate service account for this
  * Currently the same service account used to mangle user data is storing metric data, this is not best security practice.
* [x] Configure thanos to store metrics in the cloud
* [x] Turn down the data retention of prometheus from 8 weeks to a single day

Reference doc: https://docs.google.com/document/d/1oJcWY4U2lmPwBvY8qQPlP7msPc7ioVUlo3ZKkCp-i1g/edit


/cc @gitlab-com/gl-infra

Story points: 3.0
Fix syntax that will break in terraform 0.12

I tried planning with 0.12.2 on gstg, and got a bunch of errors. Luckily they are all of the form:

```
Error: Invalid attribute name

  on .terraform/modules/alerts/instance.tf line 176, in resource "google_compute_instance" "default":
 176:     "CHEF_URL"                = "${var.chef_provision.["server_url"]}"

An attribute name is required after a dot.
```

this should be easy to fix and will hopefully smooth our path to terraform 0.12.

Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6760

Story points: 1.0
PoC: Database backups based on snapshots

Postgres basebackups are currently being taken by wal-e every 24 hours. This process takes up to 16 hours and consumes a lot of IO on the primary. We cannot have wal-e take this basebackup from a replica because wal-e doesn't support that. 

During weekdays, the daily backup takes up to 16 hours to complete. It increases the load and IO wait on the primary extraordinarily when it runs during higher traffic times:

![Screenshot_from_2019-08-08_15-01-25](/uploads/a0a99dfae791cbb9e7b958ee33ebc03f/Screenshot_from_2019-08-08_15-01-25.png)

Additionally, from time to time, we run into issues with wal-e uploading to GCS. See for example https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7049.

There was an effort to use wal-g over wal-e, which would support taking basebackups from replicas. However, currently we're still waiting on a wal-g release to properly support GCS.

Even with wal-g in place, we would still have a high MTTR for restoring a full backup:
* A 3TB backup has to be fetched from GCS and extracted on the database host
* Worst case, we'd replay 24 hours worth of WAL on top of that basebackup to catch up with changes made after the basebackup started.

MTTR is likely at >= 8 hours, if not worse (it depends on how much WAL needs to be applied).

### Proposal

In order to reduce the MTTR to below 1 hour, we want to use GCE disk snapshots to take basebackups.

In more detail, we would
* Provision a dedicated replica which would consume WAL from archive and - when running - generally stay up to date with the production cluster. The replica does not participate in the HA cluster.

At regular intervals (let's say every 2 hours, but this can be tuned) we would
* Stop postgres on the replica and flush disks to ensure snapshot consistency
* Coordinate with the GCE API to grab a snapshot of the data disk
* Start postgres again to let it catch up with the upstream cluster

In order to restore a backup, we would create a new disk from the latest snapshot and instantiate a database instance from it. It would be configured as an archive replica and consume WAL from archive until the recovery point. Then, it's promoted to primary and can start serving connections.

### Benefits

* Greatly reduced MTTR: Disks can be created from snapshots in a few minutes max - no need to ship 3TB across the network
* Disk snapshots are incremental, hence cost is not a huge concern with high frequent backups
* New database instances can be created easily from the snapshots. This is also helpful if we had to recover the full cluster: We'd create all 6 instances from the same snapshot, let them catch up and then direct the replicas to talk to the chosen primary.
* Removed the nightly IO bottleneck by not taking backups from the primary any more

### Proof of concept implementation

(tbd)

### Notes

* It may be helpful if the data directory would live on its own disk and we would just snapshot that. With that, the data directory becomes a drop in we could easily mount at any database instance.

Story points: 5.0
use pushgateway to indicate pg backup start and finish

currently we capture wal-e basebackup message `Upload`  https://gitlab.com/gitlab-cookbooks/gitlab-mtail/blob/master/files/default/mtail/wale.mtail#L23 and use it in pgbasebackup alert to indicate if a base backup happened. However, this alert would never notify us if the backup failed during the process. Because of it, we missed 6 base backups in the last 14 days but never get any alert https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7049.


We should use pushgateway before and after the backup to indicate the backup start and completion instead.

Story points: 2.0
Convert the gitlab-com repo from helm template to helm local tiller

As discussed in a recent meeting, notes captured here: https://docs.google.com/document/d/1jnh5Sf9wXmok0W5iyQJS0CDItDIl5E1hQKOBwIb4w2E/edit#heading=h.ooga5woic4af

> Helm/Tiller - skarbek
> * Let’s utilize local tiller
> * Let’s convert the gitlab-com project over to utilizing local tiller
> * Create an issue

We've decided it's safer to utilize the hooks provided by helm.  To avoid the security implications by using tiller, we'll utilize a plugin that provides a local tiller.  We are currently testing successfully this method with our [monitoring repo](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/monitoring), we need to convert our [gitlab-com repo](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com) to using the same system.

/cc @jarv

Story points: 2.0
Detecting5xxForRegistry is flappy

`sum(backend_code:haproxy_server_http_responses_total:irate1m{backend="registry",code="5xx",tier="lb"}) >0`

is really flappy in the presence of sporadic errors, I believe mainly because of irate1m, irate being 'the last two samples', and over only 1m.  I think we need to re-evaluate what this is doing; I'm thinking a new synthetic 'rate5m' might be more useful to smooth it out, so we get better stronger more consistent signal when things are going wrong.  It probably needs a small number other than 0, but that's TBD; with a smoother signal, >0 may be sufficient.

/cc @cmcfarland

Story points: 1.0
deploy updated uptycs(osquery) to prod canary hosts

The SecOps team would like to proceed with rolling out an updated version of uptycs (osquery) to the production canary hosts. 

This new version has some significant performance improvements around the embedded rocksdb. So far in staging, we haven't seen any significant performance impact like we did with the previous version. 

For more info/context - see [#199](https://gitlab.com/gitlab-com/gl-security/operations/issues/199)

Story points: 2.0
Discuss/formalise recommendation for managing GKE (Google Kubernetes Engine) patching/upgrades

Following on from #6448 and the some of the implications of #6880 , we should discuss, then write down (and link from relevant places) what gl-infra expects from people spinning up GKE clusters in GCP projects.

Starting position for discussion:
1. If you're spinning up GKE clusters yourself in GCP projects (i.e. gl-infra are not directly involved beyond provisioning a project for you), then you are responsible for the maintenance of same.  gl-infra can advise/assist when explicitly requested, but will not be pro-actively looking out for things.
2. The simplest way to avoid issues with this is to turn on automatic upgrades of the GKE cluster (master + nodes).  https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades details how to do that.  This also requires that you consider uptime requirements for the thing you're running on GKE, but if it's critical to the business, perhaps gl-infra should be looking after it anyway which requires explicit discussion + handover.

Story points: 2.0
Update credentials for Salesforce used in customers.gitlab.com

Hi team, the creds for Salesforce has been recently changed and out integration is broken.

Can you please update the chef vault with the new credentials? Specifically we need to update just [two attributes](https://gitlab.com/gitlab-cookbooks/cookbook-customers-gitlab-com/blob/ca4bc07310aaf78629260425edb6d391580942ea/attributes/default.rb#L37-38). The values are stored on the `Subscription portal` shared vault from 1password, please let me know if you need access to it.

Thanks!

Closes gitlab-org/customers-gitlab-com#546

Story points: 1.0
Rollout infrastructure secrets via vault

* [ ] define a strategy for migrating secrets from GKMS / Chef vault to Hashicorp Vault
  * moving secrets over piece by piece or maybe just adapting the shim layer for secrets-management in Chef?
  * try to prevent having to live in both worlds for an extended time period
* [ ] test on one service
* [ ] rollout service group by service group

We eventually should split this up into multiple issues when we clarified the migration strategy.

Story points: 8.0
Be confident in our metrics, dashboards, and alerting related to ZFS git storage nodes

We want sufficient metrics to be able to diagnose and debug issues related to filesystems and storage on ZFS-backed git storage nodes, and our alert conditions to be satisfactory.

There are 2 sides to this:

1. ZFS-specific metrics. Check out https://github.com/ncabatoff/zfs-exporter
1. Existing metrics related to persistent disk I/O are up to scratch. Is there anything to alert on here?

From a suggestion by @andrewn.

Story points: 4.0
Investigate automating issue creation for Discourse upgrade notifications

We receive email notifications to the `ops-contact` account when a new version of Discourse is available for `forum.gitlab.com`, which are easy to ignore, lose track of, or fall victim to the [bystander effect](https://en.wikipedia.org/wiki/Bystander_effect). If possible, we should look into automating the creation of upgrade issues from these emails to more easily/consistently ensure that the notifications are received through our primary workflow (gitlab todos).

/cc @ansdval @hphilipps @cmiskell @msmiley

Story points: 1.0
Health checks failing on inactive web nodes

Reported by @andrewn via Slack: https://log.gitlab.net/goto/b6cba9be8f89b80456fd17b614f363e9. 

We put the 8 new nodes in MAINT status yesterday so that HAProxy wouldn't do health checks. But it looks like we didn't consider the fact that GLB also has health checks and reports metrics. 

Went ahead and stopped the nodes (web-29 through web-36). Next week (Week of June 17th), we will be driving a couple of different efforts around separating API traffic and also scaling pgbouncer. Depending on the outcome, if we decide to scale up web fleet again I'd like to use these 8 nodes right away. If not, we can do proper terraform MR and remove these nodes from GCP.

Story points: 1.0
Discussion: Notifications of Terraform module version bumps

When someone makes a change to a terraform module, it is no longer just `gitlab-com-infrastructure/environments/*` that needs to be updated.  We need a way to reliably notify all primary users of the module that it has been updated.

Some initial ideas are:

1. An MR template.
   - Anyone with an interest can add their name to the /cc line
2. The CODEOWNERS file
   - I'm not sure this will work in all cases
3. Update the readme with links to the places that need to be changed
   - Who is responsible for changing it? The merger? Do they just open an issue? Or submit MR's everywhere?
4. Someone's better idea

This probably effects myself, @skarbek and @Craig at first, but will eventually touch everyone in @gitlab-com/gl-infra, so the more input we can get the better. We may also want to think about doing something similar with chef.

Story points: 1.0
Update Grafana sync key

Per https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6952#note_180920686, we need to update the key used [by chef](https://gitlab.com/gitlab-cookbooks/gitlab-grafana/blob/master/recipes/export_dashboards.rb#L41) to use the updated key from 1Password in the `dashboards-sync Grafana API key` item within the `Team` vault.

Story points: 1.0