giseldo/neodataset huggingface dataset:
Add subdomain for 10k instance under testbed.gitlab.net

We need a subdomain to be created for the 10k instance:

```
10k.testbed.gitlab.net
```

The IP can be found in [dev-resources](https://gitlab.com/gitlab-com/dev-resources/-/jobs/251853108) job output under `drew-10k-lb`

See https://gitlab.com/gitlab-org/quality/performance/issues/38

Story points: 1.0
Shutdown postgres11 instances in staging

We are going to pause the postgres 11 upgrade effort just for a little while.

We should stop the postgres11 instances in staging because they're not needed (saves about 1,200 $/month).

```
 postgres11-01-db-gstg 	
 postgres11-02-db-gstg 	
 postgres11-03-db-gstg 	
 postgres11-04-db-gstg 	
 postgres11-05-db-gstg 	
 postgres11-06-db-gstg
```

Story points: 1.0
PostgreSQL: Minor upgrades from 9.6.12 to 9.6.14

We are 3 minor versions behind the latest release of PostgreSQL 9.6 and we should perform a minor upgrade.

Minor upgrades are generally safe and can be done without incurring long downtime since it just needs a restart of the postgres process. That means, we would be doing a rolling upgrade and restart across the cluster.

Minor upgrades usually fix bugs and make other non breaking improvements. 

Changelogs:
* https://www.postgresql.org/docs/9.6/release-9-6-14.html
* https://www.postgresql.org/docs/9.6/release-9-6-13.html
* https://www.postgresql.org/docs/9.6/release-9-6-12.html

Story points: 2.0
Add thanos-query from GKE to the rest of our thanos infra

Currently our GKE clusters thanos-queries are it's own datasource in grafana.  We should connect these to the ops thanos query instance such that we get metric data from all of our infrastructure.

1. [x] Write the ability to add additional query targets
1. [x] Wire it up

/cc @gitlab-com/gl-infra

Story points: 5.0
DR vs GPRD file server count differences

Currently we have 32 file servers in `dr` while `gprd` has 36 file servers.  This would seem to break the Geo feature.  Use this issue to discuss why we are missing 4 servers in the `dr` environment, and bolster our documentation to ensure that when we scale up file servers, the same occurs in `dr`.  We should also consider alerting on a situation like this as this is one of our ways to recover data.

/cc @gitlab-com/gl-infra

Story points: 1.0
Ensure that the zpool is cleanly reimported when instances are destroyed and recreated

If I GCP-destroy a ZFS-backed storage node and use terraform to reprovision it, we are confident that the correct disks will be reattached, but will the zpool be reimported after chef runs? If not, fix it!

We already know the zpool is imported and the relevant filesystem(s) mounted when the instance is rebooted.

Story points: 1.0
Increase in Postgres Dead Tuple alerts

We have seen an increase in unactionable Postgres alerts over the last on-call shift.  This always recovers on its own, but it is a change in behavior, so we should understand what is causing it. 

The alerts are:
- PostgreSQL_TooManyDeadTuples
- PostgreSQL_ReplicaStaleXmin

Dead Tuples Percentage over the last 7 days:

![Screen_Shot_2019-07-08_at_3.42.29_PM](/uploads/65f55700269e60bed022be613169ee5d/Screen_Shot_2019-07-08_at_3.42.29_PM.png)

Dead Tuple Rates over the last 7 days:

![Screen_Shot_2019-07-08_at_3.43.36_PM](/uploads/74e0659763f5c2e7d3068d5f7eb0df19/Screen_Shot_2019-07-08_at_3.43.36_PM.png)

Total Dead Tuples over the last 7 days:

![Screen_Shot_2019-07-08_at_3.56.54_PM](/uploads/d65634813fcb836d0edb5e273855d447/Screen_Shot_2019-07-08_at_3.56.54_PM.png)

Autovacuum per table over the last 7 days (note the last day or so):

![Screen_Shot_2019-07-08_at_3.44.15_PM](/uploads/92830f308f32e07553904aab39d3ea32/Screen_Shot_2019-07-08_at_3.44.15_PM.png)

All of these metrics are here:

https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m&var-environment=gprd&var-prometheus=Global&var-instance=patroni-04-db-gprd.c.gitlab-production.internal&from=1562031895380&to=1562636695380

/cc @dawsmith @Finotto

Story points: 2.0
Do we still need to provision git storage nodes to be overweighted towards one zone in a region?

Our git storage fleet is provisioned by the generic-stor terraform module, which provisions node_count nodes in the configured zone, and multizone_node_count nodes in the configured region, allocated across zones in that region by round robin. In this way file store nodes are "overweight" in the configured zone.

https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules/google/generic-stor/merge_requests/7 proposes breaking this feature of generic-stor (for reasons discussed in that MR), and implementing any required zone overweightness by provisioning 2 file store pools, 1 which specifies a zone, and 1 which specifies a region for round robin node allocation.

This issue aims to address whether we even need to do this: can we simply provision regional pools with round robin allocation, and not be overweight in any one zone?

Story points: 1.0
Consider the removal of the nginx ingress for the GKE Container Registry

With the recent implementation of the Container Registry into GKE, we went the quick route and utilized the nginx ingress provided by our helm chart in order to quickly utilize this as a PoC.  This adds a few things:
1. Complexity with the way our haproxy must transfer data to and from the GKE cluster
    * https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/commit/29e5dab8aab05d1118df46a0f84f1e47f706dd6c
    * https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/commit/b7c806f4b77ffa37e30bf429ddc295c388589568
1. Adds an extra network hop that we don't REALLY need:
    * Current: GLB -> haproxy -> registry nodes
    * Proposed: GLB -> haproxy -> nginx ingress -> pods
    * Ningx is not doing anything out of the ordinary other than forwarding traffic
1. With this configuration we are using Let's Encrypt on this new external endpoint `registry.gke.<ENV>.gitlab.<ROOT>`.  This adds a layer of configuration that we could potentially eliminate
1. This makes it hard to slowly roll traffic into either our VM's or GKE.  Instead our solution is an on/off switch
    * https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/merge_requests/159/diffs#20875b27e096b4a4356a90b6ae97d03a1dbf877a_114_117

## Proposal
1. Configure the container registry without an ingress
1. Expose the container registry service with an internal static IP that we can feed to haproxy

/cc @gitlab-org/delivery

Story points: 5.0
Dead Tuples and Stale Replica

These two errors are intermittently flapping.

Additionally, the runbook has a lot of information which is misleading or not up to date with our current environment.

```
*patroni-04-db-gprd.c.gitlab-production.internal*
 PostgreSQL dead tuples is too large

 The dead tuple ratio of import_export_uploads is greater than 5%
```

```
 *patroni-04-db-gprd.c.gitlab-production.internal*
 PostgreSQL replication slot patroni_01_db_gprd_c_gitlab_production_internal on patroni-04-db-gprd.c.gitlab-production.internal is
falling behind.


 The replication slot patroni_01_db_gprd_c_gitlab_production_internal on patroni-04-db-gprd.c.gitlab-production.internal is using
a minimum transaction ID that is 861.3k transactions old.
This can cause an increase in dead tuples on the primary. This can be
caused by long-running transactions on the master or any standby, or unused replication.
```

Story points: 1.0
git-over-SSH errors debugging

Following on from https://gitlab.com/gitlab-com/gl-infra/production/issues/844 there are still some underlying issues to resolve. In particular the regularity of issues reported in https://gitlab.com/gitlab-com/gl-infra/production/issues/844#note_187833688 suggests we could track that instance down specifically and hopefully capture enough data to find a source of at least one cause.

Current status:
Problem alleviated; some followup still to occur regarding unicorn queuing on git front-end servers

Story points: 5.0
GKE Prometheus does not store data persistently

Our prometheus instance in GKE is not storing data persistently when the pod is removed.  Despite configuring data to be stored for 4 weeks per our configurations, once the prometheus Pod is removed, an data older than the new pod is lost.  The helm chart defaults to using an EmptyDir instead of a disk.  We can solve this in 2 ways:

1. Add data persistence to the prometheus-operator
1. Finish this issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7068

I think we should pursue option 1 as there exists the potential that a pod will disappear before thanos has had a chance to grab the data and push the data into cloud storage.  **This is not a replacement for issue #7068**

/cc @gitlab-org/delivery 
/cc @bjk-gitlab

Story points: 1.0
DRY up terraform module code

With the upgrade to terraform 0.12 complete, and availability of HCL 2.0, we can now take advantage of the new language features to reduce duplication across our terraform codebase, in particular within our modules.

1. [ ] Identify near-identical modules for refactor (e.g. `generic-sv-with-group`, `generic-stor`, `generic-stor-with-group`)
1. [ ] Create/relate issues for refactor work on specific modules
1. [ ] Create/relate issues to update version pinning / deploy all changes via `gitlab-com-infrastructure` pipelines

Story points: 8.0
Redis nodes kernel upgrades

As part of the effort to alleviate saturation on Redis nodes related to https://gitlab.com/gitlab-com/gl-infra/production/issues/928, we're going to upgrade kernels on Redis nodes. @bjk-gitlab noted that the nodes are spending about 50% of their time on system, and we expect the kernel upgrades to reduce this.

@bjk-gitlab and @jarv are working on this.
Chef runs fail because user databag is not formatted

The `check` [job](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/jobs/547453) failed with:

```
$ find [a-z]* -type f -name \*json | xargs -I{} -r -n 1 bash -c 'jq . {} > /dev/null || ( echo "bad JSON {}" && exit 1 )'
$ find [a-z]* nodes -type f -name '*.json' | xargs -I{} -r -n 1 bash -c 'diff <(cat {}) <(jq . {}) > /dev/null || ( echo "You must format this file {}" && exit 1 )'
You must format this file data_bags/users/cshobe.json
You must format this file data_bags/users/yguo.json
ERROR: Job failed: exit code 1
```

Story points: 1.0
Apply vault terraform config via CI/CD

For consistency, testing and keeping vault deployment secrets in one place we should apply terraform changes for the vault GKE cluster only via CI/CD pipeline.

Other jobs that should be run via CI:
- re-keying (adding a new pgp key to the repo should trigger a re-keying and pgp encryption of all unseal keys)
- changes of the basic vault config (accounts/roles/policies)
- ...

Story points: 8.0
Add GCP provider (or alternative)  to dev-resources and deprecate DO

Recently, we've seen an increase in random errors in [dev-resources](https://gitlab.com/gitlab-com/dev-resources/) in the past couple of months. 

It has been a source of frustration for the support team when creating instances for reproducing issues or handling interviews.
* https://gitlab.slack.com/archives/C4Y5DRKLK/p1561730479035000
* https://gitlab.slack.com/archives/C4Y5DRKLK/p1561711484479900
* https://gitlab.slack.com/archives/C4Y5DRKLK/p1561564095249700
* https://gitlab.slack.com/archives/C4Y5DRKLK/p1559761492127800
* https://gitlab.slack.com/archives/C4Y5DRKLK/p1559759365125100

I don't think DO is scaling well. Some of the errors appear to be responses from DO's API (500s). We should consider [adding a GCP provider](https://cloud.google.com/community/tutorials/managing-gcp-projects-with-terraform) and start migrating off DO and maybe eventually deprecating it. 

Related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6526

Story points: 6.0
gitlab-com repo pulling master branch of our charts from pages

The gitlab-com repo pulls a version specified by an ENV variable, defaulting to `master` to determine which version of the chart we want.  Prior to switching to helm tiller we were relying on a version specified in a file: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/blob/master/GITLAB_CHARTS_VERSION

https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/blob/master/bin/.gitlab.bash#L9

Setup this repo such that we are pulling a specific version again as defined in this provided file.  Ensure that the file is updated to match that of master which is what we've been using to deploy lately.

Consider the fact that our helm charts are also running on GitLab pages.  Should pages be down for any reason, we secondary place to grab our charts.  

/cc @gitlab-org/delivery 
/cc @gitlab-com/gl-infra

Story points: 3.0
Our CI Images pull master for the helm components, consider locking these down

Both the installation of helm and the helm tiller plugin are using the latest and/or master during it's build process: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/common/blob/master/Dockerfile-ci#L5-9

Consider pinning the versions to specific releases of these to prevent issues or breakages.

Story points: 1.0
Build the staging Kubernetes Cluster for the Container Registry

Utilize this issue to track progress in creating the container registry for the staging environment.

Keep https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6862 updated to ensure we get ourselves fully documented as we proceed.

The end goal of this issue, is to have all registry traffic destined for the staging environment sent inside of the GKE platform

Story points: 3.0
Upgrade gitlab packages on postgres DR hosts

This is a proposed ~"corrective action" for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7090.

We employ two DR replicas which run on the standard omnibus package. However, we don't upgrade the package with a regular deploy and the current version is `11.2.0-rc2.ee.0`. This is because we rarely use the codebase on these hosts and only cared about postgres so far.

However, especially in the DR context, it is beneficial to have the latest codebase installed. This allows us to easily run a Rails console which talks to the delayed replica, for example. This is helpful when we recover an accidentally deleted project. It is currently not possible because the Rails console with the old codebase doesn't even start anymore on that instance.

The two DR replicas in question are (gprd here, similar in gstg):

* `postgres-dr-delayed-01-db-gprd`
* `postgres-dr-archive-01-db-gprd`

Story points: 2.0
Start rails console with read-only database session by default

This is a proposed ~"corrective action" for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7090.

We employ a "console host" on `console-01-sv-gprd` which people can use to start a Rails console (via SSH, using either the console user that starts the console automatically or by excecuting `gitlab-rails c` on the host).

### Proposal

By default, the Rails console should have a read-only database session. This allows for safe(r) use of the Rails console without risk of altering any database data accidentally. When read-write access is necessary, the session can be promoted read-write from the console.

### Implementation

In postgres, a database session can be set [read-only](https://www.postgresql.org/docs/9.6/sql-set-transaction.html) with `SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY`. When we start the Rails console, we would have to SET this attribute. This may be possible by configuring `database.yml` accordingly or by using a postgres user dedicated to console access which would have this as a default.

In order to promote a session read-write, we'd simply set it back with `SET SESSION CHARACTERISTICS AS TRANSACTION READ WRITE`. This can be shipped as part of the codebase in `Gitlab::Database` for convenience.

Story points: 1.0
Validate/Establish backup strategy

It's possible that slight "time alignment" / consistency differences between disk snapshots in a raidz array will result in an umountable zfs dataset.

If we can't restore a raidz array from GCP snapshots, then we will need to propose an alternative, i.e. by periodically shipping ZFS snapshots elsewhere from day 1 of git storage nodes on ZFS so that we do not lose our ability to recover data from several hours/days ago.

cc @T4cC0re

Story points: 8.0
Make it possible to whitelist an IP for all of GitLab.com, not just the API

Currently we are able to whitelist IPs to get around our rate limiting for the API (`/api`) but we have no way to do it for the site as a whole. In order to facilitate an internal request for whitelisting, we need to implement a way to whitelist an IP from the rate limit for the entire site.

cc/ @cmcfarland @jsalazar-gitlab

Story points: 4.0
Registry 5xx alerts are numerous with no obvious problem to resolve.

The on-call SRE's receive alerts that indicate a high number of 5xx errors from the registry service. But there are no obvious service problems (registry is down, serving requests slowly, etc.) that can be repaired. The issue often recovers before any action is taken.

Either our alerting for 5xx errors in registry are too stringent, or there is a legitimate issue with registry that needs to be fixed.

One of these should be the criteria to close this issue:
* Do we have an SLO that we must keep registry 5xx errors below a threshold? If so, are our current alerts enforcing that level, or a level higher? Create an issue(s) or merge request(s) to address the changes to our alerting.
* If the alerting is correct, what is the underlying issue? Create an issue (or issues) to resolve that problem.

Story points: 1.0
Create environment for migration testing

Create an easily re-creatable sandbox environment to test gitaly node migration from ones that are backed by ext4 to ones that are backed by ZFS.

Ideally, the deliverable will be a repository including:

* Terraform for a GitLab environment where everything is minimal except gitaly shards, where we will need a few (3?) for testing. Initially the Gitaly shards will use ext4.
* Integration with chef. Consider whether it's worth using another chef organisation or not.
* Scripts to fill the Gitaly nodes with dummy data.
* A readme explaining how everything is set up, how to use it, and how to tear it down.

Use as much resource isolation as possible, in a separate GCP project.

Story points: 4.0
Blueprint for Encrypting (TLS) internal Gitaly traffic

During the work on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6484, it became evident that we should write a blueprint and a design doc so that: 
1. Our work is well thought out in advance
2. The implementation is documented so that our customers can refer to it
3. Provides us pinning points to iterate in the future 

Will work on an MR for it.

Story points: 6.0
Service inventory catalog to product

### Background
In 2019Q1, we delivered Service Inventory Catalog ([Design Doc](https://about.gitlab.com/handbook/engineering/infrastructure/design/service-inventory-catalog/) | [Production App](https://us-central1-gitlab-infra-automation-stg.cloudfunctions.net/ui/services)) in an effort to help centralize tribal knowledge of our production services, reduce time to search for information and reduce onboarding time among other benefits. 

Given that the Service Inventory Catalog (SIC) takes a step in the direction of addressing problems that are common across teams, organizations and companies, not just GitLab, there is an interest of exploring options to take this solution to product so that our customers can run something similar for their need. 

In all, we believe that any individual or a team building service(s) (requirements, designs, ownerships, dependencies, documentations, configurations, runbooks, playbooks, security aspects) should document and maintain them. SIC aims to find a home for all of these information. Just like how it is a good practice for every version controlled project to have a README.md file, can a service-inventory-catalog become a good practice, too?

### Objective 
The objective of this issue is to keep track of the work to:
* Investigate whether it is feasible to deliver SIC to product
* Explore option(s) of how we want to deliver it (thinking from the perspective on how our customers would/might use it)
* Scope the effort and identify dependencies/resources
* Implement
* Announce

### References 
- [Service Inventory Catalog design doc](https://about.gitlab.com/handbook/engineering/infrastructure/design/service-inventory-catalog)
- [Service Catalog YML file](https://gitlab.com/gitlab-com/runbooks/blob/master/services/service-catalog.yml)
- [Team YML file](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/team.yml)
- [Deployed version](https://us-central1-gitlab-infra-automation.cloudfunctions.net/ui/)
- [Source code](https://gitlab.com/gitlab-com/gl-infra/service-catalog-app)

cc @Finotto

Story points: 8.0
Use common CI image for k8s-workloads/gitlab-com and turn on registry metrics

Now that we have a common project https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/common for the CI image and some CI config we can utilize it for `gitlab-com`. This issue tracks the following:

- [x] Use the common CI image for running pipelines in ops
- [x] Adapt pipelines so we have similar job names for applying updates
- [x] Adjust configuration so we can start scraping metrics

Story points: 2.0
Move gitlab-cookbooks group under gl-infra

# What?

Transfer the gitlab-cookbooks group to be a subgroup of gitlab-com/gl-infra on both .com and ops.

# Why?

Permissions inheritance from gl-infra, and organisation.

# How?

1. Prepare an MR to chef-repo updating all of the Berkshelf references to use the new subgroup path.
1. Move the group in .com.
1. Move the group in ops.
1. Merge that chef-repo MR.
1. Notify the infrastructure team that this has been done so that people can update their remotes. This can be done using this issue.

@gitlab-com/gl-infra what do you think of the idea? IIRC this has been discussed before. Have I missed any considerations in the "how" section?

Story points: 3.0
Fix or remove Redis role portion of shell prompt

I just noticed that our custom shell prompt is broken on the Redis servers.  The prompt always shows the string `REDIS_CHECKCMD_ERROR`.

That string is set by `/usr/local/bin/custom_ps1.sh`, which on a Redis host tries to determine the current role (primary or secondary) of the local Redis instance by connecting to the instance via Unix socket:
```
    check_cmd="$(/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket role 2>/dev/null)"
    if [[ $? -ne 0 ]]; then
        echo -n "REDIS_CHECKCMD_ERROR"
        return
    fi
    if [[ "${check_cmd}" == *"master"* ]]; then
        echo -n "PRIMARY-REDIS"
    else
        echo -n "secondary-redis"
    fi
```

However, our Redis instance isn't currently configured to accept Unix socket connections.  In `/var/opt/gitlab/redis/redis.conf` we set `unixsocketperms` but not `unixsocket` itself.  Also, Redis is configured to require authentication (`requirepass`), which `custom_ps1.sh` does not do.

Ask the SRE/DBRE team whether we should fix or remove this portion of the shell prompt.

If we decide to fix it, should we make a helper script in /usr/local/bin to successfully open a redis-cli session?  That would follow the pattern we use for psql access on Postgres hosts.  We'd also need to decide whether to re-enable Unix socket listener or use the TCP listener.  TCP is a simpler fix, since it would not require bouncing Redis.

Story points: 2.0
Investigate why API whitelist isn't working

We added a whitelist entry for @jsalazar-gitlab in [chef-repo!1280](https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1280), but when applied and reloaded it did not seem to work. 

@cmcfarland, @alejandro, and I have spent a while investigating this today and thus I decided to make an issue about it.

As of now we are in the process of trying to restart haproxy to see if that resolves the issue.

Otherwise, I'm pretty stumped.

Story points: 4.0