giseldo/neodataset huggingface dataset:
List known potential gotchas with Elasticsearch indexing of GitLab Projects

As an SRE, I want to understand the known or anticipated types of operational issue that may come with the upcoming addition of Elasticsearch indexing as a GitLab 12.1 feature, so that we can better interpret new observed behaviors or changes in workload for related subsystems.

*Background:*

While listening to today's *excellent* "Elasticsearch Deep Dive" presentation by @mdelaossa (see [video](https://drive.google.com/open?id=13wXpOL9GbZFUCw4DaRxoE_GmNjf1jDT5) and [slides](https://docs.google.com/presentation/d/1H-pCzI_LNrgrL5pJAIQgvLX8Ji0-jIKOg1QeJQzChug/edit?usp=sharing) and [notes and questions](https://docs.google.com/document/d/1cwo5n3XYaTDAJ48sMZJ8bHQVJ0RD5dlsdf28L96OZQw/edit#)), I realized a few areas where the introduction of ES indexing may significantly change the workload or scaling requirements of related subsystems (ES nodes themselves, Gitaly storage nodes, Sidekiq pools, Postgres, etc.).  And of course as will any new technology adoption, there are new failure modes to be aware of (e.g. ES node down/dead, shard redundancy, ES quorum policy for writes/reads, impact of ES availability on upstream clients, etc.).

This issue aims to collect a list of know topics for @gitlab-com/gl-infra to be aware of.  It's just for discussion and brainstorming, not for resolving any of the topics raised.

These topics may eventually be added to a runbook once they are actionable.

Story points: 3.0
Need to update pg_hba.conf for version/license/customers restored backups

Currently the data team is pulling data from restored backups of the version/license/customers DBs. We are going to start pulling from a different kube cluster and are getting an error around the pg_hba.conf. We need this to be updated so that we can start pulling from the new cluster. Just let us know what you need to know from us and we'll provide/help with whatever we can

cc @tayloramurphy

Story points: 2.0
Support different versions of terraform in our terraform CI/CD tests

Currently, our tf_format test uses a global terraform version specified at the root of the repository for it's lint tests. For some earlier versions of ```terraform fmt```, this is ok. But the newest versions of terraform format in such a way that those files fail using the specified older global version.

Since each environment can have it's own version of terraform specified, we should update our tests to use that specific version. This will allow us to use newer versions without having to drag the entire repo up into new versions of terraform.

Story points: 2.0
Update `Query source` to `Global` in Grafana dashboards that are not pulling any metrics

We have dashboards like this: https://dashboards.gitlab.net/d/_MKRXrSmk/pull-mirrors?orgId=1&refresh=30s&from=now-7d&to=now&fullscreen&edit&panelId=13 where the data source isn't populating anything. 

Per Slack discussion: https://gitlab.slack.com/archives/CB3LSMEJV/p1560355187114000, if we change the `Query` source to `Global` it works. 

See below Before & After comparison. 

BEFORE
![grafana_dashboard](/uploads/b819524fa0977840bd4fa42c4789695f/grafana_dashboard.png)

AFTER
![grafana_dashboard_after](/uploads/4cc9fd7c03ec289ec7a1236a754af209/grafana_dashboard_after.png)

Story points: 7.0
Add generic Linux observability tools to our hosts

As an SRE or DBRE, I want our Linux hosts to include tools for ad hoc observation, so that I can collect short-term metrics and investigate behaviors that are impractical or out of scope for our general purpose monitoring.

Background:
Prometheus provides a variety of metrics collected periodically.  This serves us well for most purposes.  However, sometimes we need more granular or narrowly scoped instrumentation for ad hoc investigations.

Examples:
* Polling disk I/O statistics at 1-second intervals is helpful when analyzing suspected bursts of I/O contention that get smoothed over at less frequent polling intervals.
* Measuring variation in memory access latency on a VM that runs an in-memory data store (e.g. Redis) can reveal an otherwise opaque root cause to transient query response time spikes.
* Measuring trends in TCP connection open/close events can help inform tuning the kernel's TCP stack to more gracefully handle traffic spikes.  It can also lead to improving our general purpose diagnostic monitoring, to alert us when approaching saturation of certain finite resources (e.g. TCP connection table, pool of available client ports, etc.).

In this Issue, let's build a wish list of tools we'd like to have available on our Linux hosts.  To start us off, we have this list from the Slack discussion:
* iostat: Per block device I/O statistics, including queue depth, %busy, mean read/write latency, etc.
* sysstat: Provides "sar" utility, which gives wide variety of usage statistics, system-wide or for specific PIDs.
* [linux-tools](http://www.brendangregg.com/perf.html): Lightweight tracing facility, used via the perf-suite tools (perf, perf-top, perf-mem, perf-trace, perf-ftrace, etc.), with eBPF support for recent kernels.
* iftop: "top" for network flows, showing which remote IPs are currently using the most network throughput.
* ifstat: "vmstat" for network interfaces, polling network throughput on each interface.

Story points: 2.0
Investigate git storage nodes using more disk space than expected

Some of our Gitaly storage nodes have a significant gap between actual disk space used versus what our "project_statistics" table reports.

Try to find what dominates this discrepancy.  Time-box the discovery effort, as we are soon planning to significantly change our git storage architecture.  This investigation aims to understand the nature and magnitude of the measurement error, but it mostly matters if it's caused by a systematic accounting error that can be improved upon.

Example:
Host: file-25-stor-gprd (a.k.a. "nfs-file25")
Actual disk space used: 14 TB
Reported disk space used: 12 TB

For more background on (re)discovering this discrepancy, see:
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6796#note_179818651

Story points: 3.0
Update build-runner s3 settings before they lose support in version 12.0 of GitLab runner.

We need to updated the build runner cache settings to the new format before 12.0 is rolled out to the runners. Specifically someone needs to go in and change the vault secrets.

The build runners for the omnibus packages are throwing deprecation warnings on startup about their use of the runner cache s3 parameters. The configuration they are using is deprecated and being removed in 12.0

An MR for updating the visible settings is here: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1208

But the access secrets are stored in vault and also need to be updated.

They need to be moved from where they are to be nested one level further down under the `s3` key.

Story points: 1.0
Update consul to v1.5.1

Our current version ([1.0.3](https://gitlab.com/gitlab-cookbooks/gitlab_consul/blob/d94d678ab95f41d2528f1a032e430410e43bd854/attributes/default.rb#L1)) doesn't allow for multi-tag querying, which we'll need for example to select a canary api node (which has tags `"cny"`, and `"api"`). That feature was introduced on v1.3.0, but there's no impediment for getting on the latest version (some breaking changes, but none that affect us. See https://github.com/hashicorp/consul/blob/master/CHANGELOG.md).

/cc @dawsmith

Story points: 1.0
Cleanup Chef ACL Permissions

In working to grant `read only` access to Chef resources it was discovered that our default permission model was incredibly wide, such that anyone who was a 'user' in the Chef server could do anything except add another user (delete nodes, update roles, update environments, etc).

This needs to be cleaned up and locked down so that by default users have no access in the `gitlab` chef organization unless granted it.

Story points: 6.0
Camo proxy: secure configuration

From @msmiley on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6836#note_178684472:
We should discuss rate limiting, concurrency, and request timeouts for Camo's proxied requests. And we should decide whether to apply those limits at Camo's ingress load balancer or at each Camo instance's egress. I don't yet have a well-formed threat model, but off the cuff, we want to avoid making Camo attractive as an anonymizing proxy for attackers (abuse via us against a victim's domain) while also preventing attackers from too easily saturating our pool of Camo instances (DoS against us).

Story points: 3.0
Packagecloud Upgrade REQUIRED

**Our version of packagecloud, 2.6.0, will stop working on the 24th of June.**

This is due to AWS deprecating support for the API in which packagecloud is utilizing.  An upgrade is available.  We should upgrade before this due date.

The email I received:

>>>
AWS S3 is deprecating the AWS S3 SigV2 API on June 24, 2019 [1], which packagecloud:enterprise 2.0.6 and earlier currently use. New versions of packagecloud:enterprise (versions 2.0.7 and higher) use AWS CloudFront for serving package objects, instead of using AWS S3 directly. This upgrade significantly improves package download speeds and avoids the S3 API deprecation.

To continue using packagecloud:enterprise after June 24th, 2019, you will need to upgrade to version 2.0.8, which includes support for CloudFront. On packagecloud:enterprise version 2.0.8, you can optionally enable or disable CloudFront.

Upgrading to packagecloud:enterprise 2.0.8 will allow you to create a CloudFront distribution and test it before AWS deprecates the S3 API in a few weeks.

Note that our next release of packagecloud:enterprise, version 3.0.0, will require CloudFront. 

Creating a new CloudFront distribution can take 30 minutes or longer. We strongly urge all customers to upgrade to 2.0.8 now so that they can create their CloudFront distributions while still serving packages from AWS S3 before the deprecation takes place. This upgrade will allow for a seamless switch-over with no downtime.

Waiting to upgrade to packagecloud:enterprise until June 24, 2019, will almost certainly result in downtime as you will need to wait for the creation of the CloudFront distribution to enable downloading of package objects.

Users should follow the upgrade instructions: https://packagecloud.atlassian.net/wiki/spaces/ENTERPRISE/pages/15269926/Upgrading

Once the 2.0.8 upgrade is complete, users should follow the CloudFront setup instructions:
https://packagecloud.atlassian.net/wiki/spaces/ENTERPRISE/pages/501972993/AWS+CloudFront+Setup

Contact support@packagecloud.io if you have any questions or problems.

Happy packaging!

[1]: https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#UsingAWSSDK-sig2-deprecation
>>>

/cc @gitlab-com/gl-infra 
/cc @gitlab-org/distribution

Story points: 4.0
Giving read-only access to chef

We have ~~2~~ 3 access-requests open for read-only access to chef.

1. https://gitlab.com/gitlab-com/access-requests/issues/958
1. https://gitlab.com/gitlab-com/access-requests/issues/286
1. gitlab-com/access-requests#1110

Gathering this into an issue.
Looks like we could do:
https://github.com/chef/knife-acl#create-read-only-group-with-read-only-access

@northrup appears to have made a read-only group, but we need to do the other knife-acl commands to finish making up the group and test with @pharrison and @asaba

Story points: 3.0
New S3 bucket for Greenhouse data extract

We're going to be using the BI connector for Greenhouse to get data from them. They do nightly CSV dumps to S3.

I need a bucket with read/write access. `datateam-greenhouse-extract` as a name would be :thumbsup: 

I'll just need the Access Key and Secret Key

@ahanselka could you help with this?

Story points: 3.0
Secrets Management - Vault Design Document

Review and make any required changes to the design document on [Vault](https://about.gitlab.com/handbook/engineering/infrastructure/library/vault/) and add it to the [engineering/infrastructure/design/](https://about.gitlab.com/handbook/engineering/infrastructure/design/) section of the handbook. 

You may be required to ask questions about the document. That exploration and research should be considered part of this issue. Please close this task only once the design is agreed upon, the link is public, and work is ready to begin.

Story points: 3.0
Benchmark syslog vs logging_collector

Configs:

```
echo "
log_destination = syslog
logging_collector = off
log_statement = none
log_min_duration_statement = 0
" > pg.syslog.conf

echo "
log_destination = stderr
logging_collector = off
log_statement = none
log_min_duration_statement = 0
" > pg.stderr.collector_off.conf

echo "
log_destination = stderr
logging_collector = on
log_statement = none
log_min_duration_statement = 0
" > pg.stderr.collector_on.conf

echo "
log_destination = csvlog
logging_collector = off
log_statement = none
log_min_duration_statement = 0
" > pg.csvlog.collector_off.conf

echo "
log_destination = csvlog
logging_collector = on
log_statement = none
log_min_duration_statement = 0
" > pg.csvlog.collector_on.conf

echo "
log_destination = stderr
log_statement = none
log_min_duration_statement = -1
" > pg.no_logs.conf
```

Story points: 1.0
DBRE sync meeting scheduling

The DBRE sync was scheduled for Tuesdays, 2.30pm-3pm UTC. We have a conflict with the Secure&Defend team meeting which starts at 2.45pmUTC and should re-schedule the DBRE sync call.

As pointed out, a later time would suit US timezones better. Let's see what time/day would work best.

cc @gl-database @dawsmith @ansdval

Story points: 0.0
Benchmarks for relevant candidate configs

Benchmark the following configurations:

All:
  - n1-standard-32
  - ubuntu xenial
  - `4.15.0-1033-gcp` kernel
  - ZoL 0.7.5

raidz1:
  - 9 SSD zpool
  - 2 local SSD L2ARC

raidz1 HDD:
  - 9 HDD zpool
  - 2 local SSD L2ARC

single:
  - 1 SSD
  - 2 local SSD L2ARC

ext4:
  - 1 SSD ext4

Story points: 2.0
Camo proxy monitoring, alerting, and runbooks

Camo proxy will need some basic monitoring beyond VM health.  Starting points requiring further thought/expansion
1. Service responsive via load-balancer - pageable
1. Service responsive on each node - alert, but don't page

May require a pre-prepared encoded URL that is requested to ensure end-to-end functionality; would suggest something static on gitlab.com itself like a logo/image that won't change URL ever, and that we can expect to be up in the cases that we need camo proxy to be working (i.e. we won't be alerting because 'The Internet' is down, and if gitlab.com is completely down, we won't notice this alert specifically in the storm of alerts)

Ideally we need some sort of graphing of the throughput/usage, although it's unclear at this stage how this will be implemented (mtail? custom exporter?)

In addition to alert response runbooks, we'll need to include debugging steps for verifying correctness and determining where a request is failing; it seems likely that we'll end up responding to requests asking why a URL isn't responding.

Story points: 2.0
Create GCP Projects for Groups

There have been several requests recently to create GCP projects for various groups to do work in.  Currently, these projects are being created ad-hoc with no standardization in how they are set up.

The https://ops.gitlab.net/gitlab-com/group-projects project has been created to standardize this process and give each group an isolated place to work without having to go through the infra team for each request.

Advantages of this include cost tracking, and greater accountability on each team.  It also reduces the workload on the infrastructure team for non-production work.  By empowering teams to manage their own isolated sandboxes in a safe way, everyone benefits.

The following list will be updated as this issue evolves to include the projects which will be created initially.

| Project  |  Admin |
|---|---|
| geo  | Rachel Nienaber - rnienaber@gitlab.com  |
| release  | Darby Frey - dfrey@gitlab.com |
| verify  | Elliot Ruston - erushton@gitlab.com |
| customer-success  | Joel Krooswyk - jkrooswyk@gitlab.com |
| professional-services| Daniel Peric - dperic@gitlab.com |

Story points: 5.0
Whitelist Atlasssian IP address Space for API Calls

Atlasssian is getting `429` rate limiting from the front-end HA Proxy nodes for API access.

This is to whitelist their requested infrastructure so that they can have an integration with `gitlab.com`

Atlassian provides a link for IP Addresses for Atlassian Cloud (https://ip-ranges.atlassian.com/?_ga=2.49521830.1829182039.1559668865-157908062.1559668865) we will use this to derive the content for our whitelist:

```
http https://ip-ranges.atlassian.com/?_ga=2.49521830.1829182039.1559668865-157908062.1559668865 | jq '.items[].cidr'
```

Story points: 3.0
Alert when zpool is running low on disk space

This issue is a placeholder and needs work.

SREs should receive a non-paging alert when some threshold (to be determined) of zpool space is reached. Be mindful about how the reservation filesystem interacts with this. The threshold chosen should be low enough that we are unlikely to run out of disk space before the awake on-call picks up the non-paging alert.

Consider adding a predictive alert (i.e. using prometheus' `predict_linear`) to send an alert when we are predicted to run out of zpool space in a week.

Consider adding a paging alert at a higher, emergency threshold. Discuss this with the team.

Make sure grafana dashboards are in-place and working to inspect zpool utilization.

Story points: 2.0
Figure out how to deal with registry service memory issues on kubernetes

Currently we have a job named registry-restarter that runs once a day on the VM fleet.

I'm wondering how this will work exactly in kubernetes, I assume the service will crash and restart but the reason why we have this restarter job is to prevent errors. It does this by carefully draining each node from haproxy and restarting the service.

* daily restart job https://ops.gitlab.net/gitlab-com/gl-infra/registry-restarter/pipelines
* registry memory https://dashboards.gitlab.net/d/bd2Kl8Imk/registry-host-stats?orgId=1&refresh=1m&fullscreen&panelId=7&from=now-7d&to=now

![Screen_Shot_2019-06-04_at_3.04.29_PM](/uploads/76065557671869c1f56be91cce87873c/Screen_Shot_2019-06-04_at_3.04.29_PM.png)

Story points: 1.0
k8s-workloads uses a direct download from a third party, this should be built into an image

If the third party site is down for any reason, we may have a failed job unrelated to changes being tested in the pipeline.  We should instead mirror/fork the project and create an image for us to utilize to prevent ourselves from being susecptable to such a failure.

Reference conversation: https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/merge_requests/1#note_26510

Story points: 1.0
Replicas missing in staging patroni cluster

Currently, there's only two instances in the patroni cluster in gstg:

```
root@patroni-02-db-gstg.c.gitlab-staging-1.internal:/var/log/gitlab/postgresql# gitlab-patronictl list
+---------------+------------------------------------------------+---------------+--------+---------+-----------+
|    Cluster    |                     Member                     |      Host     |  Role  |  State  | Lag in MB |
+---------------+------------------------------------------------+---------------+--------+---------+-----------+
| pg-ha-cluster | patroni-02-db-gstg.c.gitlab-staging-1.internal | 10.224.29.102 | Leader | running |         0 |
| pg-ha-cluster | patroni-04-db-gstg.c.gitlab-staging-1.internal | 10.224.29.104 |        | running |           |
+---------------+------------------------------------------------+---------------+--------+---------+-----------+
```

On other nodes, postgres is not running currently. This may be related to the network issue https://gitlab.com/gitlab-com/gl-infra/production/issues/862.

Story points: 1.0
Decide on initial ZFS configuration

After https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6726 is complete, we need to propose an initial set of config options for ZFS on git storage nodes.

Current questions we need to answer (please add to this list, it's by no means complete):

1. Amount of memory to dedicate to ARC (we have 120GB total)
1. Size of reservation filesystem in the zpool
1. zfs recordsize
1. Use native compression? If so, what algorithm? Investigate the disk space savings using real git repositories.
1. Use native encryption? (requires the new ZoL 0.8.0, see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6777)
1. Should we use quotas for the repository filesystem, e.g. 16TB? This might help us get a sense of when a node is getting full, exclusive of snapshot bulk.

Please use the discussion threads below to keep comments digestable.

~~The output of this issue doesn't need to be code, but instead can be a comment that we reference when it comes to writing infra code.~~ Since gitlab_disks already exists, let's send MRs to that as part of this issue. This configuration is what we intend to use on a single canary in production, and refine from there.

See also https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6306

Story points: 2.0
New machines can be bootstrapped with either ZFS or ext4 mounted filesystems

Git storage nodes are just one of many stateful node classes that are instantiated using the terraform generic-stor module: https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-stor.

All of our machines, stateful or not, run a version of the bootstrap script found in https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/bootstrap. Among other things, this script detects GCP PDs, formats them with ext4, and mounts them at a configurable location.

Our new git storage nodes will use ZFS instead of ext4, and will possibly have many PDs in a raidz configuration depending on the output of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6726.

Write a chef cookbook containing a recipe that idempotently formats and mounts PDs in a configurable way: variables passed in by the terraform module should instruct the recipe as to which filesystem / disk config to use. A ZFS config and ext4 should be supported initially. With regards to ZFS, it's fine if we only support the vdev config we have already agreed on: i.e. only raidz1 or only single disk.

Write a new iteration of the bootstrap script that does not format and mount persistent disks.

Add this recipe to the beginning of each stateful node's chef run list. It must not interact pathologically with existing bootstrap scripts that **do** format and mount disks.

After this issue is closed I should be able to dial up the count of the new storage nodes declared in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6727, and see a usable ZFS filesystem mounted at /var/opt/gitlab.

The recipe, if no config is injected, should behave as it does today: format a single PD at /dev/disk/by-id/google-persistent-disk-1 with ext4 and mount it up.

For an existing non-git stateful role, i.e. redis, I should be able to do each of the following steps sequentially without causing data loss or downtime:

1. Add the new recipe to the role's runlist.
1. Bump the bootstrap script version to the new one that mounts nothing.
1. Roll out new nodes.

Try these things in staging before production.

The main benefits of moving this functionality to chef are improved testability and speed of iteration. The former is particularly important as we would roll this change out to all stateful nodes.

Note that the bootstrap script also detects, formats, and mounts log disks. Moving this functionality to chef is not essential, but it may be worth moving it if everything mounting-and-formatting-related appears simpler in one place.

Story points: 4.0