giseldo/neodataset huggingface dataset:
Grafana Dashboard git sync broken

The [grafana-dashboards](https://gitlab.com/gitlab-org/grafana-dashboards) repo is synced by the [gitlab-grafana::export_dashboards](https://gitlab.com/gitlab-cookbooks/gitlab-grafana/blob/master/recipes/export_dashboards.rb) recipe.

The account/secret config is currently broken, so the sync has been failing.

Story points: 2.0
chat-ops access for .com support

Hi there! The .com support team would love to be able to use chat-ops commands.

The following people have active `ops` accounts and are ready to be added to the appropriate group to allow access. 
Would it be possible to get them going?

- [x] @tatkins 
- [x] @tristan 
- [x] @namhokim 
- [x] @arihantar 
- [x] @j.collins 
- [x] @deandre 
- [x] @amandarueda 
- [x] @jeromeuy 
- [x] @anazir 
- [x] @cwainaina 

Thanks!

Story points: 1.0
Consider additional alerting added for our storage nodes

Recently our storage nodes surpassed 80% usage on the 4 new nodes that has been spun up.  So we spun 4 new nodes up.  Now we need alerting to prevent surprises in the future.

Let's set up an overall alert if any file server breaches 80% full, warn us.   These servers may be subject to our standard alerting policy for file storage, but the process of remediation is much slower and has the potential to muck with a lot of the community.  We should consider a dedicated alert for these systems.

https://dashboards.gitlab.net/d/W_Pbu9Smk/storage-stats?refresh=30m&orgId=1

This has the consequence of alerting us today until we rebalance the git repos on our nodes. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5360

Story points: 1.0
gitlab-restore-bot was missing from postgres restore project

restore pipeline seems to start failing from 1/8 11:32pm est
https://gitlab.com/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/pipelines
It's due to this line https://gitlab.com/gitlab-com/gl-infra/gitlab-restore/postgres-gprd/blob/master/common.sh#L19 error out on `{"message":"403 Forbidden"}` 
I found out gitlab-restore-bot  is no longer a member of the the project, so I added it back as a developer and expiration date at 2033 as temporary fix. Not sure what happened to this account and whether my fix is correct.

@abrandl  please review and let me know

Story points: 0.0
Investigate Consul vs. Etcd for service discovery

The Q1 OKR's include operationalizing service discovery for GitLab.com.

Current documents all refer to Consul as the service discovery mechanism. In the Geo Group Conversation this morning, @sytses had some [good points](https://gitlab.com/gitlab-org/gitlab-ee/issues/3789#note_129839448) about Etcd starting to look like the industry standard. Since we are moving towards being cloud native eventually, we will eventually be using Etcd as part of Kubernetes, and at that point we will have two solutions to support.

Consul is easier to use, and it is what we are currently using, but surely there is some value in supporting a single solution rather than two.

The Geo team is currently researching this issue (https://gitlab.com/gitlab-org/gitlab-ee/issues/8932). The @gitlab\-com/gl\-infra  team should also look into it before work starts, and either document why Consul will be the better solution long term, or lay out a path to convert existing consul use to etcd.

Story points: 1.0
ChatOps SQL explain unable to connect to pgbouncer

The ChatOps SQL explain does not work at the moment: https://gitlab.slack.com/archives/C101F3796/p1546962452337200

The error is:

```
/app/vendor/bundle/ruby/2.4.0/gems/pg-1.1.3/lib/pg.rb:56:in `initialize': ERROR:  pgbouncer cannot connect to server (PG::ConnectionBad)
```

This can be reproduced with any `/chatops run explain SELECT...` query.

As pointed out on Slack, the [CI variables](https://ops.gitlab.net/gitlab-com/chatops/settings/ci_cd) control which host to connect to. Currently, the CI job connects to `10.217.4.2` which is `pgbouncer-03-db-gprd.c.gitlab-production.internal`.

We may want to change this setting to a more stable hostname, preferrably one that points to a replica.

Story points: 1.0
Runners on Windows for GitLab Runner Team

## Description

Create a new windows environment for https://gitlab.com/gitlab-org/gitlab-runner/issues/3755 https://gitlab.com/gitlab-org/gitlab-runner/issues/3757 and have the latest version of GitLab runner installed.

## Proposal

We would need a **windows machine** with the following software installed; **Docker** & **gitlab-runner** running as shell executor with powershell so that we can start [building the containers](https://gitlab.com/gitlab-org/gitlab-runner/issues/3755), and [running tests](https://gitlab.com/gitlab-org/gitlab-runner/issues/3757). GCP [provides windows machine](https://cloud.google.com/compute/docs/instances/windows/) which has the following [pricing](https://cloud.google.com/compute/pricing#windows_server_pricing) which costs need to be calculated. This machine should not be provisioned manually as it would be second class from the beginning since all our linux environment is already automated. We can use a mixture of terraform and chef to provision and configure the machine.

As pointed out in https://gitlab.com/gitlab-org/gitlab-runner/issues/3757#note_129070759 we need at least Windows server Core 2016, but we can go with [Windows Server, version 1803](https://cloud.google.com/compute/docs/instances/windows/#windows_server) since it's the latest version of windows.

The following software needs to be installed on the machine

- Openssh
- docker
- gitlab-runner
- Git https://github.com/git-for-windows/git

## Links to related issues and merge requests / references

- https://gitlab.com/gitlab-org/gitlab-runner/issues/3755
- https://gitlab.com/gitlab-org/gitlab-runner/issues/3757

Story points: 5.0
Require that all scripts performing maintenance have a dry-run output that can be reviewed

This has come up multiple times in incidents and I think its worth making it a bit more formal that if maintenance is automated there should be a dry-run mode that displays exactly what the script is doing, ssh commands, api calls, etc. so that it can be reviewed. 

Corrective action for https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5874

Story points: 1.0
Unresolvable service discovery config spamming Sentry

See https://sentry.gitlab.net/gitlab/staginggitlabcom/issues/612471:

```
IPAddr::InvalidAddressError: invalid address
  from ipaddr.rb:649:in `in6_addr'
  from ipaddr.rb:586:in `initialize'
  from gitlab/database/load_balancing/service_discovery.rb:139:in `new'
  from gitlab/database/load_balancing/service_discovery.rb:139:in `nameserver_ip'
  from gitlab/database/load_balancing/service_discovery.rb:130:in `resolver'
  from gitlab/database/load_balancing/service_discovery.rb:103:in `addresses_from_dns'
  from gitlab/database/load_balancing/service_discovery.rb:68:in `refresh_if_necessary'
  from gitlab/database/load_balancing/service_discovery.rb:41:in `block (2 levels) in start'
  from gitlab/database/load_balancing/service_discovery.rb:38:in `loop'
  from gitlab/database/load_balancing/service_discovery.rb:38:in `block in start'
Gitlab::Database::LoadBalancing::ServiceDiscovery::UnresolvableNameserverError: could not resolve localhost
  from gitlab/database/load_balancing/service_discovery.rb:144:in `rescue in nameserver_ip'
  from gitlab/database/load_balancing/service_discovery.rb:138:in `nameserver_ip'
  from gitlab/database/load_balancing/service_discovery.rb:130:in `resolver'
  from gitlab/database/load_balancing/service_discovery.rb:103:in `addresses_from_dns'
  from gitlab/database/load_balancing/service_discovery.rb:68:in `refresh_if_necessary'
  from gitlab/database/load_balancing/service_discovery.rb:41:in `block (2 levels) in start'
  from gitlab/database/load_balancing/service_discovery.rb:38:in `loop'
  from gitlab/database/load_balancing/service_discovery.rb:38:in `block in start'
```


The config is:

```
  load_balancing: {"hosts":[],"discover":{"record":"replica.patroni.service.consul.","nameserver":"localhost","port":8600}}
```

The error suggests this local nameserver isn't able to resolve that record.

Story points: 1.0
Create @ groups to reduce gl-infra spam

Currently, folks are using @gitlab\-com/gl\-infra  to draw attention to all infrastructure issues.  This is a very large group, and there are a lot of these alerts.  Not everybody needs to see them.

I propose splitting out these types of tags into smaller groups.  It would be easier for everyone to maintain focus if they could only be tagged on issues that they have an interest in.

We could start by splitting out the 3 new SRE groups, or we could create one for each functional area.

Story points: 2.0
Chef automation breaks when deleting files

When deleting a file from the chef repo, knife tries to push the deleted file to the chef server.  This causes the whole pipeline to fail.

If there are other changed files, they will not be uploaded to the repo, because just doing another commit won't work.  Only changed files are pushed, so on the subsequent commit, that file will no longer be changed.

We need to filter out deleted files from the list of changed files to upload to the chef server.

This MR addresses the issue: 
https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/503

Story points: 1.0
Fluentd repos are broken on gitter hosts

While updating the package cache on gitter hosts today, all returned the following error

```
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://packages.treasuredata.com trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 901F9177AB97ACBE

W: GPG error: https://deb.nodesource.com trusty InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 1655A0AB68576280
W: Failed to fetch http://packages.treasuredata.com/2/ubuntu/trusty/dists/trusty/InRelease

W: Some index files failed to download. They have been ignored, or old ones used instead.
```

This is due to an update to the GPG key for fluentd package repositories. We need to download and import the new key (reference the manual process [here](https://www.fluentd.org/blog/update-gpg-key-for-td-agent))

---

Some discussion, https://gitlab.slack.com/archives/CB3LSMEJV/p1546890404268400

Story points: 3.0
Ensure integrity of terraform changes before apply/merge

[RCA](#5813) corrective action:

> Use [Atlantis](https://runatlantis.io) for automating and queuing terraform deployments

We should implement [Atlantis](https://runatlantis.io) for distributed locking/queuing across merge requests, and incorporate that into the work for &9.

The description for this issue was originally `Setup Atlantis for Terraform deployments on ops instance`. More fundamentally, this issue is intended to ensure the integrity of all terraform changes before running apply and merging to master. Utilizing Atlantis (or Terraform Enterprise) would be one way of implementing a FIFO queue for merge requests, but we may also be able to leverage [merge trains](https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.html) within the GitLab product, itself.

Story points: 5.0
Clean up orphaned resources in Gitter environment

The following DNS entries in the `prod.gitter` [zone](https://console.aws.amazon.com/route53/home?region=us-east-1#resource-record-sets:ZLV6X5P28OJXS) reference non-existent resources; we should ensure that we have any/all associated infrastructure removed from terraform and configuration cleaned up from ansible, as well as removing any corresponding DNS records/resources in the beta environment.

```
consul-01.prod.gitter
consul-02.prod.gitter
consul-03.prod.gitter
master-01.prod.gitter
master-02.prod.gitter
master-03.prod.gitter
minion-01.prod.gitter
minion-02.prod.gitter
packer.prod.gitter
redis-03.prod.gitter
vpn-01.prod.gitter
```

Story points: 1.0
Remove unused EBS volumes in Gitter AWS account

A recent review of EBS volumes showed the following 13 volumes as unused/unattached. If they are no longer needed, we should delete the volumes.

![Screen_Shot_2019-01-03_at_3.01.42_PM](/uploads/4d4017beb5d23bd8b0158aff30f37ece/Screen_Shot_2019-01-03_at_3.01.42_PM.png)

/cc @MadLittleMods @andrewn

Story points: 1.0
Setup NAT Gateways for all Gitter AZs

As a follow-up to #5492 and gitlab-com/gl-infra/production#559, we introduced a NAT gateway for egress from internal hosts within the Gitter environment, instead of the legacy VPN which also acted as a NAT instance.

The new gateway is currently serving traffic for all subnets, but is susceptible to disruption if Amazon suffers an outage of the Availability Zone where the NAT Gateway is provisioned, and/or traffic is disrupted between AZs.

Now that the issue from gitlab-com/gl-infra/production#559 has been mitigated, we need to shore up the infrastructure, and provision additional NAT gateways in each availability zone, and associate the local private subnets in those AZs.

Story points: 2.0
Setup autoscaling for Gitter bastion

As a follow-up to #5492 and gitlab-com/gl-infra/production#559, we introduced a bastion host for access to the Gitter environment, instead of the legacy VPN. The new bastion is a single instance, with no redundancy for fault-tolerance or HA as a quick mitigation to gitlab-com/gl-infra/production#559, but now we need to shore up the infrastructure.

1. [ ] Setup a Network Load Balancer for SSH traffic to the bastion node
1. [ ] Setup an Autoscaling group for bastion nodes
1. [ ] Setup a Launch Configuration / Launch Template for bastion nodes
1. [ ] Provision a deterministic host key on bootstrap for all bastion nodes (allows clients to rely on strict host key checking)

Story points: 5.0
Database Reviews

* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8497#note_124026675
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8927#note_126726752
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8669#note_126158494
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19740#note_125938200
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23098#note_125777941
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24056#note_128145058
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24042#note_127617989
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8497#note_127501661
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/issues/51854#note_127186345
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8949#note_127338816
* [x] @abrandl trigram https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23952#note_128130394
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18425#note_126834567
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8961
* [x] @abrandl Thu https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21499
* [x] @abrandl Thu https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2694
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8988#note_128739113
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24119
* [x] @abrandl binary join https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8977#note_128700496
* [x] @abrandl trigram https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23952#note_128621830
* [x] @abrandl LB https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8961
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21499 with https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7238
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24047#note_128914394
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8949#note_128914150
* [x] @abrandl rather heavy migration https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8669
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18425/diffs#note_129308895
* [ ] Fixtures https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8240#note_129069979
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24144
* [x] High risk https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2694#note_144679
* [x] N+1 https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2480
* [x] @abrandl DB load balancing https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9071#note_130145897

No or later milestone:

* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23464#note_126871390
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/issues/54643#note_125483243

Story points: 0.0
Set up DB replica for analytics pulls into EDW

Organization:
1. In this issue, we want to setup access to the archive replica and configure it such that it can be used from the ELT loads
1. In a related issue https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5970, we are going to provide network-level access to said replica for the ELT runner jobs (with appropriate firewall rules).

Discussion summary:
1. The ELT load will only need read-only permissions (hence a replica is fine).
1. No strict SLA guarantees regarding uptime/availability/performance.
1. Queries from the ELT load are expected to finish within 5 minutes. Statement timeouts will be enforced accordingly.



-------


Our current method for getting updated production data into our warehouse is as follows:

* Update pseudonymizer config if possible https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/pseudonymizer.yml
* Ping someone on production team to run the pseudonymizer (usually Stan)
  * Hope that nothing fails or bombs out writing CSVs to GCS
* Import generated CSVs from object storage into Snowflake

This presents a number of challenges, not the least of which is precious time from a member of the production team every time we want updated data. 

Reading through https://about.gitlab.com/handbook/engineering/infrastructure/database/disaster_recovery.html and https://gitlab.com/gitlab-restore/postgres-gprd it seems like we should be able to set up an isolated replica that is updated on a regular basis with production data. (We do this regularly for version, customer, and license dbs https://gitlab.com/gitlab-restore/version.gitlab.com#vpc-setup) We don't need up to the minute updates, delays of 1-2 weeks is fine.

If we follow the same model where only our project's runner is whitelisted for access I think it'll simplify the security footprint and risk. We'd still have an explicit list of fields we'd be pulling that security would approve for updates and increased data pulls.

My time request is that something be stood up by the end of January. We've got some OKRs related to modeling of the dotcom data. 

@stanhu does this seem reasonable? I know it'd save you some time. 

@dawsmith thoughts on this? Is this something @yguo could setup?

@jeremy @tszuromi just FYI - I'm trying to close the time between updates for production data. 

cc @tlapiana

Story points: 2.0
Investigate solutions to vacuuming our database prior to failover

During a recent failover https://gitlab.com/gitlab-com/gl-infra/production/issues/637, patroni-04 was elected leader.  Upon becoming the leader, we received an alert that we've exceeded our thresholds for Dead Tuples.

![image](/uploads/70b247f384080c84eaa52d8630ad98ee/image.png)

This node started off with high percentages of dead tuples.  

Use this issue to investigate and resolve the following questions:
1. Why do secondaries not have the same data or analyze table information for dead tuples as the primary?
1. Should this alert be considered high priority?
1. If this is a problem what can be done prior to failovers to ensure potentially elected secondaries do not have a high percentage of dead tuples?

Story points: 2.0
Improve WAL-E alert for 'WALEBackupDelayed'

During our recent failover https://gitlab.com/gitlab-com/gl-infra/production/issues/637, the alert `WALEBackupDelayed` fired.  This was technically a false alarm.  Wal-E backups continued to work, the data that prometheus was looking at was correct, but fired when another server had begun processing uploads to S3.

![image](/uploads/bedcb6f07cc4ab04bf1e82e1f74a197b/image.png)

https://prometheus.gprd.gitlab.net/graph?g0.range_input=2d&g0.expr=gitlab_com%3Alast_wale_backup_age_in_seconds%20&g0.tab=0

Improve this alert to reduce alert fatigue during incidents.  This alert introduced noise into an already stressful situation.

Story points: 1.0
Doing graceful DB failovers under the new Patroni setup

Right now Patroni failovers can result in 50x errors during the short period of demoting a DB a promoting another.

We can leverage the fact that all client connections go through pgbouncer and that pgbouncer can be [paused](https://pgbouncer.github.io/usage.html#pause-db) from accepting new connections. However, with the current setup we can't utilize the pausing feature. Consider this scenario:

1. `patroni-01` is a master, we want to failover to `patroni-02`
1. We pause pgbouncer on `patroni-01`, existing client connections are completed and closed
1. Clients are trying to establish new connections, they are still forwarded to `patroni-01` by the ILB but pgbouncer isn't accepting any yet
1. We do a failover to `patroni-02`
1. We resume (un-pause) pgbouncer on `patroni-01`
1. Now `patroni-01` has connections that are trying to execute read/write queries, they will fail

We need to pause connections at the level of the ILB before they are forwarded to a master, but this doesn't seem possible. I tried marking all of Patroni ILB backends as unhealthy to see if new connections will fail immediately or will wait for a backend to become healthy, but the ILB forwarded the connection to one of the replicas! (even when its backend is marked as unhealthy)

So I think we may need to have a dedicated pgbouncer node in front of the ILB, so we can control the connection flow at a higher level. pgbouncer on such node will be configured with the ILB FQDN, so we don't run into all the issues of using Consul watchers to update the databases configuration. We can go one step further and drop the ILB altogether in favor of using Consul DNS record for the master node (assuming https://gitlab.com/gitlab-com/gl-infra/production/issues/633 has made it to production)

Story points: 3.0
Large file in generic-sv-with-group terraform module

There is a 42M file in the [generic-sv-with-group](https://gitlab.com/gitlab-com/gl-infra/terraform-modules/google/generic-sv-with-group) terraform module at:

`.terraform/plugins/darwin_amd64/terraform-provider-google_v1.18.0_x4`

On a fast connection, it doesn't make much difference - but with the slow internet provider problems that I have been having, it's been preventing me from running `tf-init`.  I suspect that it is slowing down other people as well, and they're not noticing because it's not failing.

I didn't delete it because I wasn't sure if it was there for a reason.

Does this need to be there? And can we delete it from the repo?

As a side note, these repo's probably all need .gitignore and README.md files

Story points: 1.0