giseldo/neodataset huggingface dataset:
Register services necessary for using consul for host inventory

As part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6153 and based on the notes of https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20889, have each node register its services with consul for the purpose of node inventory.

- chef-repo MR: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1199

/cc @dawsmith @devin @jarv

Story points: 5.0
Help debug SSO Enforcement by running a few lines in rails console in staging

## Why

I'm testing SSO Enforcement on staging and it is working for groups but not a particular project. 

## What

On staging, run the following from a rails console:

```ruby
Gitlab::Auth::GroupSaml::SsoEnforcer.define_method(:active_session?){ false }
policy = ProjectPolicy.new(User.find_by(username: 'jamedjo'), Project.find_by_full_path('jej-group-saml-test/test-subproject-within-enforced-ssso'))
policy.debug(:read_project)
policy.debug(:guest_access)
policy.lookup_access_level!
policy.needs_new_sso_session?
```

Story points: 1.0
Can terraform git storage nodes that use ZFS

After https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6726 is complete, implement terraform code that will allow us to provision "new" file servers with the decided hardware configuration. These new file servers should be a separate TF resource declaration with their own `count`.

The MR should not actually cause any new nodes to be provisioned (i.e. set count on the new resource declaration to 0). Old storage nodes should not be deleted(!).

Story points: 4.0
Decide on disk layout for storage nodes

Following the results from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6306, decide on a storage layout for vdevs. The 2 main contenders at the moment are:

* RAIDZ1 with 9 disks (8 usable, 1 parity)
* Single disk

Single disk is often called out as bad practice for ZFS, but [GCP PDs](https://cloud.google.com/persistent-disk/) are advertised as having high availability, data redundancy, and error correction. We need to evaluate this and weigh up whether it's worth adding redundancy at the ZFS level.

The throughput of the configurations must also be a factor. We don't want reduced throughput compared to today.

The total usable filesystem space per node must be at least 16TB (overlaps with discussion with quotas in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6850), the same as we have today. Note that this will be less than the zpool space due to a reservation filesystem (see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6850), and this zpool space may be less than the total PD disk space provisioned due to RAIDZ redundancy (if we choose raidz).

We intend to keep the same shard size as today. At the time of writing this is 32 x 16TB in prod. Changing this would involve changing too many things at once.

Another thing to take into consideration is that the zpool space may need to be substantially larger than what we expect due to snapshot [bloat after a git repack operation](https://christian.amsuess.com/idea-incubator/space-efficient-git). A ZFS snapshot taken before the repack will reference disjoint blocks to one taken after. Therefore the zpool disk usage for the repo will be doubled until the old snapshot passes out of the retention window. If repacking occurred simultaneously for every repo on a GitLab installation (which is unlikely), our filesystem usage would spike to double. Therefore in the worst case we would need at least twice the usable filesystem space as the data we intend to store on the node. However, since repacking of different repos should be spread out in time in realistic scenarios, we could use a lower multiple.

Also, decide on a storage layout for L2ARC. There is not much debate about this at the time of writing. The initially proposed configuration is 2 x 375GB ephemeral [local SSDs](https://cloud.google.com/compute/docs/disks/local-ssd) (they are fixed size) connected via NVMe.

We have already decided to keep using n1-standard-32 VMs for the storage nodes, so memory and CPU will not change.

The deliverable of this issue is not necessarily code, as we will need to tackle this very early in the project, but  a comment that we can use for reference when we come to write some terraform.

Performance characteristics of the candidate configurations are here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6883

Finally, satisfy yourself that we don't make life too hard for ourselves when it comes to restoring an old backup from GCE snapshots if we choose raidz.

Story points: 1.0
Grant db-geo access also to Toon, Douglas & Ash

My apologies, according to the [process](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/granting-rails-or-db-access.md#process) the requests at https://gitlab.com/gitlab-com/access-requests/issues/804 and https://gitlab.com/gitlab-com/access-requests/issues/803 should have made here, but thanks for granting the access anyway!

But additionally, I'd also like to request access to the Geo tracking database on DR. I tried to connect, but failed.

```
❯ ssh to1ne-db-geo@dr-console
to1ne-db-geo@console-01-sv-dr.c.gitlab-dr.internal: Permission denied (publickey).
```

Unix usernames: `to1ne` and `dbalexandre`

Story points: 3.0
Extend source port range of network ACL to fix #6656

## What are we going to do?
Change the port range in Inbound ACL rule 700 of production-external-network-acl Network ACL to "1024-65535"

## Why are we doing it?
Right now, roughly 10% of outbound TCP connections fail, when the randomly chosen source port is between 1024 and 6665 (the current lower limit of the port range); the reply (e.g. SYN-ACK) packets are dropped by the Network ACL we want to change.  See #6656 for the report + diagnosis of the issue.

## When are we going to do it?

* Start time: 2019-05-21 08:00 UTC
* Duration: 1 minute
* Estimated end time: 2019-05-21 08:01 UTC

## How are we going to do it?
Manual edit of the ACL rules using the AWS Web Console

## How are we preparing for it?
Co-ordinating with Gitter engineers and GitLab SRE's

## What can we check before starting?
Nothing

## What can we check afterwards to ensure that it's working?
Use the debug one-liner from #6656 (`for i in $(seq 1 20); do curl -vi api.ipify.org; done`).  We're expecting that all these calls succeed, quickly.  Any failures indicate the fix has not had the desired effect; more than 10% failures indicate we've inexplicably made the situation worse.

## Impact

* Type of impact: none expected
* What will happen: Nothing negative
* Do we expect downtime? (set the override in pagerduty): None

## How are we communicating this to our customers?
* Nothing required

## What is the rollback plan?
Change the lower bound of the source port back to 6665

Story points: 1.0
Update *.gstg.gitlab.net certificate

We are seeing some odd issues with the internal load balancer and we noticed a failing certificate. We should update this.


- pre-conditions for execution of the step
1. The newly renewed certificate has been downloaded with sslmate. ```sslmate download *.gstg.gitlab.net```
2. The proper fields to be updated in the gkms vaults are identified and the old cert is verified to match the new one. In the chef-repo, run these commands:
* ```./bin/gkms-vault-show frontend-loadbalancer gstg```
* ```./bin/gkms-vault-show gitlab-omnibus-secrets gstg```
3. A backup copy of the old certificate field is stored locally.
4. A properly formatted version of the new cert is already made and formatted for JSON.
```awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' *.gstg.gitlab.net.chained.crt > json.*.gstg.gitlab.net.cert```

- execution commands for the step
1. Stop chef on the service fleet that serves the cert in question. ```knife ssh "role:gstg-base-lb-fe" “sudo service chef-client stop“```
2. Edit the vaults that contain the cert using a command like this: ```./bin/gkms-vault-edit frontend-loadbalancer gstg```
3. Find and replace the cert field I identified earlier in the JSON and save the changes. The following fields will be updated: ```gitlab-haprox -> ssl -> internal_crt``` 
4. There is a load balancer certificate in GCP that needs to be updated also. This [certificate](https://console.cloud.google.com/net-services/loadbalancing/advanced/sslCertificates/list?project=gitlab-staging-1&authuser=0&organizationId=769164969568&orgonly=true&sslCertificateTablesize=50) will be deleted and replaced with the new certificate.
5. Force a chef-run on a server like fe-01-lb-gstg.c.gitlab-staging-1.internal and verify it is using the new certificate.
6. Cert verification ```echo | openssl s_client -connect 10.224.14.18:443 2>/dev/null | openssl x509 -noout -dates```
7. Restart chef on the nodes from the first step. ```knife ssh “role:gstg-base-lb-fe” “sudo service chef-client start”```

- Rollback
1. Edit the altered vault using the gkms-vault-edit command.
2. Replace the changed cert with the old one that was copied into the .crt file.
3. Delete and replace the GCP certificate with a backup of the old certificate recorded in the pre-steps.
4. Save the changes and force a chef run on the test system and verify it is fixed (or back to normal).

Story points: 1.0
GitLab Hosted version of Codesandbox Sandpack

### Overview

As part of gitlab-org/gitlab-ce#58548 we need to setup a new S3 bucket to serve the static sandpack script required for a GitLab hosted Codesandbox.

### Proposal

Serve the javascript from an S3 bucket on a custom domain with SSL enabled. Requirements:
 - Domain (needs to be purchased): gitlab-sandbox.com
 - S3 Bucket Configured
 - SSL Enabled

The entirety of the linked the feature is targeted for the 12.1 Release (2019-07-22) and this is a dependency to that.

### Links / References

https://docs.gitlab.com/ee/user/project/web_ide/index.html#enabling-client-side-evaluation
https://gitlab.com/gitlab-org/gitlab-ce/issues/58548#note_170759249

Story points: 3.0
Setup salesforce Omniauth with GitLab.com

With https://gitlab.com/gitlab-org/gitlab-ce/issues/57077#note_169752576, we are being asked to setup Omniauth for GitLab.com with Salesforce.  There is a promotion where we are looking to go live by May 29 so we have a clear understanding of expected dates.  For infra, we should look to have an initial setup running next week (week of May 20).

Documentation:
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27834/diffs?commit_id=97fa2bf90da6489e84b143f4d7b00395b634ffc7

### Todo

 - [ ] Disable Salesforce oauth sign-in source on staging, for consistency with other providers

Story points: 2.0
Update Salesforce's sandbox credentials  for customers.stg.gitlab.com

We need to update the Sandbox [credentials](https://gitlab.com/gitlab-cookbooks/cookbook-customers-gitlab-com/blob/5c4ce40bdec1e9752be6650213471caca4647b3a/templates/default/secrets.yml.erb#L48-52) for Salesforce, new credentials ara available in the `Subscription Portal` shared vault from 1password.

Story points: 1.0
Missing haproxy logs since the 1.8 upgrade

We are currently not receiving or logging anything for haproxy to `/var/log/haproxy.log`, the last log message I see is on `May  2 21:29:07`


This roughly corresponds to the haproxy upgrade:

```
Commandline: apt-get -q -y install haproxy=1.8.8-1ubuntu0.4
Requested-By: alejandro (1007)
Install: haproxy:amd64 (1.8.8-1ubuntu0.4)
End-Date: 2019-05-02  21:43:12
```

cc @alejandro

Story points: 2.0
ChatOps: Match statement timeout with production setting

ChatOps can currently only get plan for queries that return within 5s. That is because we have a lower `statement_timeout` set than the default production setting (which is 15s).

This means that we cannot use ChatOps for rather expensive queries. Oftentimes, those are the queries we actually care about and want to improve and hence we need plans for them.

Currently, the procedure is to reach out to somebody with database access to get plans for those queries (example: https://gitlab.slack.com/archives/C3NBYFJ6N/p1557140206109800).

The proposal here is to match the production `statement_timeout` setting for ChatOps, such that we can get plans through chatops for queries up to 15s.

Story points: 1.0
Set lifecycle policy to change the storage class of some objects in the log-archive bucket

The log archive bucket is 484TB and represents a significant cost when it comes to object storage.
This bucket alone is 10k/month, the same amount of data in coldline storage is about 10% of this cost. 
This issue is to discuss a policy for changing the storage class of objects to coldline storage.
Before we can work on this issue we need to decide on the lifecycle policy. 

I suggest that we start with anything older than 90days.


## References

- https://cloud.google.com/storage/docs/lifecycle
- https://cloud.google.com/storage/docs/storage-classes

Story points: 2.0
Update cert for registry.gitlab.com

# Production Change - Criticality 2 ~"C2"

| Change Objective                                       | Prevent registry.gitlab.com certificates from expiring.|
|:---|:---|
| Change Type                                            | ~C2 |
| Services Impacted                                      | GitLab.com |
| Change Team Members                                    | @cmcfarland |
| Change Severity                                        | ~S2 |
| Buddy check or tested in staging                       | I don't think we can test this in staging. Will pair with a colleague for coverage. |
| Schedule of the change                                 | **See Comments** |
| Duration of the change                                 | **See Comments** |
| Detailed steps for the change. Each step must include: | See below |

* Verify certificate is valid locally.
* Re-format the certificate for vault entry ( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/ssl_cert.md )
* Use gkms-vault-edit to replace the SSL certificate.
* Find some way to verify the change is working in production.

Story points: 1.0
Update cert for gitlab.com

# Production Change - Criticality 2 ~"C2"

| Change Objective                                       | Prevent gitlab.com certificates from expiring.|
|:---|:---|
| Change Type                                            | ~C2 |
| Services Impacted                                      | GitLab.com |
| Change Team Members                                    | @cmcfarland |
| Change Severity                                        | ~S2 |
| Buddy check or tested in staging                       | I don't think we can test this in staging. Will pair with a colleague for coverage. |
| Schedule of the change                                 | **See Comments** |
| Duration of the change                                 | **See Comments** |
| Detailed steps for the change. Each step must include: | See below |

* Verify certificate is valid locally.
* Re-format the certificate for vault entry ( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/ssl_cert.md )
* Use gkms-vault-edit to replace the SSL certificate.
* Find some way to verify the change is working in production.

Story points: 1.0
Update cert for staging.gitlab.com

# Production Change - Criticality 2 ~"C2"

| Change Objective                                       | Prevent staging.gitlab.com certificates from expiring. |
|:---|:---|
| Change Type                                            | ~C3 |
| Services Impacted                                      | Staging.GitLab.com |
| Change Team Members                                    | @cmcfarland @devin |
| Change Severity                                        | ~S2 |
| Buddy check or tested in staging                       | I don't think we can test this in staging. Will pair with a colleague for coverage. |
| Schedule of the change                                 | **See Comments** |
| Duration of the change                                 | **See Comments** |
| Detailed steps for the change. Each step must include: | See below |

* Verify certificate is valid locally.
* Re-format the certificate for vault entry ( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/ssl_cert.md )
* Use gkms-vault-edit to replace the SSL certificate.
* Find some way to verify the change is working in production.

Story points: 1.0
Standard database console access to archive replica for analytics

Follow-up from https://gitlab.com/gitlab-com/access-requests/issues/835#note_164280462

To my knowledge, a standard (read-only) database console would give access to a production replica (ie one that participates in the Patroni HA cluster) only.

I would suggest we implement access for the archive replica for cases like https://gitlab.com/gitlab-com/access-requests/issues/835. This would allow people to run adhoc queries with much more freedom (timeouts) and no concerns about affecting the production HA cluster for GitLab.com (but with the same data).

Story points: 2.0
Update fe-lb hosts to Ubuntu Bionic

Part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6539

Unblocks https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5293

### pre

- chef-repo MR: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1050
- gitlab-com-infrastructure MR: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/752

### gstg

- chef-repo MR: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1062
- gitlab-com-infrastructure MR: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/753

### gprd

- chef-repo MR: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1065
- gitlab-com-infrastructure MR: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/767
- Production change issue: https://gitlab.com/gitlab-com/gl-infra/production/issues/815

Story points: 4.0
Update cert for *.gprd.gitlab.net in production

# Production Change - Criticality 2 ~"C2"

| Change Objective                                       | Prevent *.gprd.gitlab.net certificates from expiring.|
|:---|:---|
| Change Type                                            | ~C2 |
| Services Impacted                                      | GitLab.com |
| Change Team Members                                    | @cmcfarland |
| Change Severity                                        | ~S2 |
| Buddy check or tested in staging                       | I don't think we can test this in staging. Will pair with a colleague for coverage. |
| Schedule of the change                                 | **See Comments** |
| Duration of the change                                 | **See Comments** |
| Detailed steps for the change. Each step must include: | See below |

* Verify certificate is valid locally.
* Re-format the certificate for vault entry ( https://gitlab.com/gitlab-com/runbooks/blob/master/howto/ssl_cert.md )
* Use gkms-vault-edit to replace the SSL certificate.
* Find some way to verify the change is working in production.

Story points: 1.0
Pull Mirrors not running in production

PullMirrorsOverdueQueueTooLarge Alert is Firing and Pull Mirror jobs aren't being processed.

![Screen_Shot_2019-04-23_at_4.11.57_PM](/uploads/38c18ed9789e9b528e5f7afa4b2ca6fc/Screen_Shot_2019-04-23_at_4.11.57_PM.png)
![Screen_Shot_2019-04-23_at_4.12.56_PM](/uploads/0624dc0892167f22b24d965bd127e635/Screen_Shot_2019-04-23_at_4.12.56_PM.png)

Following the runbook caused mirror jobs to run again for a moment, but they stopped again soon afterward

Story points: 2.0
Rebalance Git Nodes - File 26 and File 28 are above 80%

We should look at File-26 and File-28 to rebalance and figure out growth on File-28.
https://dashboards.gitlab.net/d/W_Pbu9Smk/storage-stats?refresh=30m&orgId=1

file-28 is at 87% and had been steadily growing over the last month.

Story points: 1.0
ChatOps access request for @lmcandrew on ops.gitlab.net

Similar request as https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6454

## What

I'd like to be given chatops access so I can enable feature flags. I think this entails being added to a group on ops.gitlab.net

## Why

In the Manage team we are using per-group feature flags for the first time, and I'd like to test both the features themselves and the process of enabling feature flags.

Story points: 1.0
DR gitaly jobs show as not up in prometheus

Using this query `up{tier="stor"}` on https://prometheus.dr.gitlab.net should show `1` on the lines for `job="gitaly"`, just like it is with the others.

The services are running on the server:
```
file-01-stor-dr.c.gitlab-dr.internal:~$ sudo gitlab-ctl status
run: gitaly: (pid 17958) 337468s; run: log: (pid 2383) 862976s
run: logrotate: (pid 19323) 2565s; run: log: (pid 2381) 862976s
```

Story points: 1.0
Important SSL certificates expiring in May, 2019

Opening up this issue so we have extra attention on the ssl certificates that are expiring in may, some over contribute:

* gitlab.com *May 11 23:59:59 2019 GMT*
* int.gprd.gitlab.net *May 11 23:59:59 2019 GMT*
* registry.gitlab.com *May 22 23:59:59 2019 GMT*

Story points: 1.0
Omnibus version incorrect in chef

The omnibus version value is not correct in some environments.  After deploying `11.10.0-rc8.ee.0`, the roles look like this:

Production:
```
chef_type:           role
default_attributes:
  omnibus-gitlab:
    package:
      key:     ccc8474ea719d51d16542f1a69117ec7d7465bd18b19a832
      name:    gitlab-ee
      repo:    gitlab/pre-release
      use_key: true
      version: 11.10.0-rc8.ee.0
description:         THIS IS A PLACEHOLDER FOR TAKEOFF - VERSION IS NOT VALID, DO NOT UPLOAD TO CHEF
env_run_lists:
json_class:          Chef::Role
name:                gprd-omnibus-version
override_attributes:
run_list:
```

Staging also appears correct:
```
chef_type:           role
default_attributes:
  omnibus-gitlab:
    package:
      key:     ccc8474ea719d51d16542f1a69117ec7d7465bd18b19a832
      name:    gitlab-ee
      repo:    gitlab/pre-release
      use_key: true
      version: 11.10.0-rc6.ee.0
description:         THIS IS A PLACEHOLDER FOR TAKEOFF - VERSION IS NOT VALID, DO NOT UPLOAD TO CHEF
env_run_lists:
json_class:          Chef::Role
name:                gstg-omnibus-version
override_attributes:
run_list:
```

DR (Note the lack of version):
```
chef_type:           role
default_attributes:
  omnibus-gitlab:
    package:
      key:     ccc8474ea719d51d16542f1a69117ec7d7465bd18b19a832
      name:    gitlab-ee
      repo:    gitlab/pre-release
      use_key: true
description:         THIS IS A PLACEHOLDER FOR TAKEOFF VERSION IS NOT VALID, DO NOT UPLOAD TO CHEF
env_run_lists:
json_class:          Chef::Role
name:                dr-omnibus-version
override_attributes:
run_list:
```

Pre was updated before production:
```
chef_type:           role
default_attributes:
  omnibus-gitlab:
    package:
      key:     ccc8474ea719d51d16542f1a69117ec7d7465bd18b19a832
      name:    gitlab-ee
      repo:    gitlab/pre-release
      use_key: true
      version: 11.9.8-ee.0
description:         THIS IS A PLACEHOLDER FOR TAKEOFF - VERSION IS NOT VALID, DO NOT UPLOAD TO CHEF
env_run_lists:
json_class:          Chef::Role
name:                pre-omnibus-version
override_attributes:
run_list:
```

Story points: 3.0
ChatOps access request for @nfriend on ops.gitlab.com

### What
I'd like to be given ChatOps access in order to enable and disable feature flags on staging.gitlab.com.

### Why
Being able to enable and disable feature flags on staging.gitlab.com will allow me to test features hidden behind feature flags as part of our QA process.

### Background

I'm following the example of this issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6454

Story points: 1.0