giseldo/neodataset huggingface dataset:
Database Access For Dylan Griffith

The configure team regularly finds ourselves needing to query the database for certain things like:

- debug production incidents
- understanding how many rows we have in certain tables (how this might affect migrations)
- analytics on usage of our product
- query the DB to understand the impact of introducing new constraints in the DB

We don't have anybody in our team that can execute these queries so we usually rely on others to do this for us but sometimes like in the case of analytics on product usage we need to do slightly exploratory querying to know what we're looking for. It's hard to do this without prod DB access since we do one `COUNT` query and this tells us what kinds of other queries we might want to do to understand the data better.

Could you please provide access to @DylanGriffith so that the Configure team is better setup to manage these kinds of problems in future without external support.


Story points: 1.0
Reconfigure the Gitaly cgroup cookbook to match new file server nodes

Gitaly currently uses the default cgroup settings (32GB of memory, 1024 CPU shares, etc) from the Gitaly cgroup cookbook.

However, the machines Gitaly is now running on on GitLab.com are much bigger, with much more memory and we should ensure that we're using the optimal settings.

```
root@file-01-stor-gprd.c.gitlab-production.internal:/home/andrewn# free -g
              total        used        free      shared  buff/cache   available
Mem:            118           2          83           0          31         113
Swap:             0           0           0
```

113GB of unused memory on file-01 :disappointed: 

```
root@file-01-stor-gprd.c.gitlab-production.internal:/home/andrewn# cat /sys/fs/cgroup/memory/system.slice/gitlab-runsvdir.service/memory.max_usage_in_bytes
34359738368
```

Gitaly cgroup constrained to 32GB of memory, using a maximum of ~20% of available RAM on dedicated Gitaly machines:

![image](/uploads/2026203e3d7668e210688d6511d540c3/image.png)

<sup>https://dashboards.gitlab.net/d/Zy6xM95mk/incident-405-failing-api-health-checks?panelId=12&fullscreen&orgId=1&from=now-24h&to=now</sup>

cc @tommy.morgan @jacobvosmaer\-gitlab

Story points: 2.0
fewer requests are going to web-01 than other VMs in the load balancing pool

It appears web-01 in gprd is responding much faster than the other instances as you can see in this graph:
https://dashboards.gitlab.net/d/HyOiXrSmz/rails?refresh=5m&panelId=2&fullscreen&orgId=1

The reason, however, is that fewer requests are being routed to web-01 resulting in the liveness check skewing the latency results. From this there are a couple open questions:

- Should we be excluding liveness from our latency measurements?
- Why is web-01 receiving fewer requests relative to liveness?

Out of the last 1000 requests on web-01 524 were liveness, on web-02 354 were liveness.

Story points: 1.0
Create a wildcard certificate for `*.gitlab-review.app`

We currently use a self-signed certificate for `*.gitlab-review.app` (for EE review apps) but it seems to not work at all.

I get the following when visiting https://gitlab-review-improve-re-ffvbep.gitlab-review.app/ in Firefox:

>>>
The owner of gitlab-review-improve-re-ffvbep.gitlab-review.app has configured their website improperly. To protect your information from being stolen, Firefox has not connected to this website.

This site uses HTTP Strict Transport Security (HSTS) to specify that Firefox may only connect to it securely. As a result, it is not possible to add an exception for this certificate.

gitlab-review-improve-re-ffvbep.gitlab-review.app uses an invalid security certificate.

The certificate is not trusted because it is self-signed.

The certificate is only valid for ingress.local.

Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT
>>>

We should have a proper wildcard certificate for `*.gitlab-review.app`.

/cc @marin

Story points: 2.0
Create VM/Cloud inventory first iteration

https://gitlab.slack.com/archives/CB3LSMEJV/p1533834709000494

After getting some quetions - we should have a tool that produces some kind of inventory of hosts and IPs to help us track down things among our cloud providers.  This will also help new people find things.

Before doing - come up with a plan - is there an existing tool we can use?
Do we just want output to go to a spreadsheet?
Get a 2nd set of eyes to review the plan and then go.

Story points: 5.0
New Storage Servers

We will need to create new storage servers soon. We are approaching the 60% mark on 17, 18, 19, and 20. This should perhaps obviously be done post migration.

- [X] Build new storage servers (https://gitlab.com/gitlab-com/gitlab-com-infrastructure/merge_requests/497)
- [x] Add storage servers to chef-repo so that they are mounted by NFS
- [x] Ensure the servers are set up in `gitlab.rb` to be storage options
- [x] Attempt to migrate a test project to the new servers
- [x] Update the default repository locations in the GitLab admin interface.

Story points: 1.0
create alert and dashboard for sidekiq exceptions

We currently have very little visibility into this and it would be nice to add this to our sidekiq stats dashboard.

* azure - https://performance.gitlab.net/d/000000154/sidekiq-stats?refresh=5m&orgId=1
* gcp  - https://dashboards.gitlab.net/d/000000154/sidekiq-stats?refresh=5m&orgId=1

During the last failover rehearsal we saw a large number of exceptions we should alert on. https://dashboards.gitlab.net/d/l8ifheiik/geo-status?panelId=33&fullscreen&orgId=1&from=1533223456362&to=1533250079043

Story points: 3.0
WAL-e wal-push should not overwrite existing data in S3

I'm wondering if the wal-e `archive_command` overwrites existing WAL segments in S3 - we should make sure this doesn't happen (as per [docs](https://www.postgresql.org/docs/9.6/static/continuous-archiving.html), `archive_command` should fail in this situation):

> The archive command should generally be designed to refuse to overwrite any pre-existing archive file. This is an important safety > feature to preserve the integrity of your archive in case of administrator error (such as sending the output of two different servers to the same archive directory).

/cc @NikolayS

Story points: 1.0
if any of the registry/api/web backends go unhealthy, the entire site goes down

We have three load balancers that serve gitlab.com in GCP, https, git and http.
These three load balancers are serving traffic to haproxy VMs that have the following health checks that validate the health of their corresponding backends:

```
frontend check_ssh
    bind 0.0.0.0:8003
    mode http
    option splice-auto
    acl no_be_srvs_ssh nbsrv(ssh) lt 1
    monitor-uri /-/available-ssh
    monitor fail if no_be_srvs_ssh

frontend check_http
    bind 0.0.0.0:8001
    mode http
    option splice-auto
    acl no_be_srvs_web nbsrv(web) lt 1
    monitor-uri /-/available-http
    monitor fail if no_be_srvs_web

frontend check_https
    bind 0.0.0.0:8002
    mode http
    option splice-auto
    acl no_be_srvs_web nbsrv(web) lt 1
    acl no_be_srvs_api nbsrv(api) lt 1
    acl no_be_srvs_reg nbsrv(registry) lt 1
    monitor-uri /-/available-https
    monitor fail if no_be_srvs_web || no_be_srvs_api || no_be_srvs_reg
```

the last check means that if either web, api, or registry has zero healthy backend hosts we will cease to serve https traffic. 


* This configuration puts an unnecessary coupling between the registry and the rest of the site as since the registry is served from registry.gitlab.com we could have a dedicated load balancer for it.
* git https traffic is routed to the git vms, we should probably move git https traffic to a different fleet or use the web fleet.

Story points: 8.0
Migrate registry S3 bucket to GCS

As part of https://gitlab.com/gitlab-com/migration/issues/635, we should also standardize the registry bucket name. This is likely more work than https://gitlab.com/gitlab-com/migration/issues/635 and thus should be done migration~4038690 . However, the bucket itself will be added in https://gitlab.com/gitlab-com/gitlab-com-infrastructure/merge_requests/440.

Story points: 4.0
Architecture diagrams to aid in systematic debugging

Per 2018-07-05 failover

The debugging of https://gitlab.com/gitlab-com/migration/issues/642 was made unnecessarily difficult due to us having to puzzle out the exact CDN setup. We should ensure up-to-date architecture diagrams exist for at least gprd, documenting the full route taken for all variations of all presented TCP services, so we can debug these things systematically when they come up.

/cc @dawsmith @jarv

Story points: 1.0
Decide on what endpoint to use for the HAProxy health check

* In azure we use `/` which goes all the way to rails and serves a 302.
* In GCP we use `/-/liveness` which also hits the database.

There was some discussion in https://gitlab.com/gitlab-com/infrastructure/issues/4481 on whether this is a good idea and that maybe we should use a health check that does not depend on the database.

Story points: 1.0
Re-enable archiving mode on postgres nodes in GCP

They are being disabled in https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2207.

Why? Quoting [Slack](https://gitlab.slack.com/archives/C9W4C89LY/p1530204627000477):

> My concern is, if we used a single bucket for Azure and GCP, failover demos would push WAL segments with different timeline, and I think that would confuse WAL-E when we failback because it constantly looks for newer timelines to fetch segments for

After we failover permanently, we should re-enable them.

Story points: 1.0
Fix reverse DNS for GCP IPs

Noted in the 2018-06-21 failover

Various IPs on the GCP side are missing appropriate reverse DNS.

For instance, `gprd.gitlab.com` resolves to `35.231.145.151`, which has reverse DNS of `151.145.231.35.bc.googleusercontent.com`

Since post-failover, we want `gitlab.com` to point to this IP, we should ensure the reverse DNS is set to `gitlab.com`

We don't send email via these IPs, so it's not super-important, but I do think it's desirable.

Other IPs that matter will include the ones that `registry.gprd.gitlab.com`, `gprd.gitlab.io` and `altssh.gprd.gitlab.com` (doesn't exist yet) will resolve to. Perhaps I'm missing others as well.

We should consider setting appropriate rDNS for IPs in gstg as well

Story points: 1.0
repmgr should be added to centralized logging

We currently do not have these included and they probably should be. 

Example:
```
2018-04-26_02:31:41.69640 [2018-04-26 02:31:41] [INFO] connecting to database 'host=postgres-01.db.prd.gitlab.com port=5432 user=gitlab_repmgr dbname=gitlab_repmgr'
2018-04-26_02:31:41.69733 [2018-04-26 02:31:41] [ERROR] connection to database failed: could not connect to server: Connection refused
2018-04-26_02:31:41.69749 	Is the server running on host "postgres-01.db.prd.gitlab.com" (127.0.0.2) and accepting
2018-04-26_02:31:41.69756 	TCP/IP connections on port 5432?
2018-04-26_02:31:41.69764
2018-04-26_02:31:42.70236 [2018-04-26 02:31:42] [INFO] connecting to database 'host=postgres-01.db.prd.gitlab.com port=5432 user=gitlab_repmgr dbname=gitlab_repmgr'
2018-04-26_02:31:42.70289 [2018-04-26 02:31:42] [ERROR] connection to database failed: could not connect to server: Connection refused
2018-04-26_02:31:42.70303 	Is the server running on host "postgres-01.db.prd.gitlab.com" (127.0.0.2) and accepting
2018-04-26_02:31:42.70310 	TCP/IP connections on port 5432?
2018-04-26_02:31:42.70316
```

Story points: 2.0
consul client and consul cluster should be added to centralized logging

We currently do not have these included and they probably should be. I think we can simply add these to the existing postgres index.

## Consul client

Unfortunately these logs are unstructured:

```
/var/log/gitlab/consul/current

example:
2018-06-20_00:07:26.61126     2018/06/20 00:07:26 [INFO] serf: EventMemberFailed: postgres-02.db.prd.gitlab.com 10.66.1.102
2018-06-20_00:07:27.59971     2018/06/20 00:07:27 [ERR] agent: Failed to invoke watch handler '/var/opt/gitlab/consul/scripts/failover_pgbouncer': exit status 4
2018-06-20_00:07:32.90036     2018/06/20 00:07:32 [INFO] serf: EventMemberJoin: postgres-02.db.prd.gitlab.com 10.66.1.102
2018-06-20_00:09:22.54212     2018/06/20 00:09:22 [ERR] agent: Failed to invoke watch handler '/var/opt/gitlab/consul/scripts/failover_

```

## Consul server

Logged using the systemd journal, we don't currently have a fluentd plugin for this so it would need to be added.
https://github.com/reevoo/fluent-plugin-systemd

Story points: 2.0
Monitor consul members

We've had instances of postgres secondaries missing from consul in https://gitlab.com/gitlab-com/infrastructure/issues/4415 and https://gitlab.com/gitlab-com/infrastructure/issues/4330.

It would be useful to have prometheus check and alert on e.g. the member count for "postgres", so we can detect that situation automatically.

There is a consul exporter available: https://github.com/prometheus/consul_exporter

Story points: 1.0
Add centralized logging to prometheus and grafana hosts

When troubleshooting https://gitlab.com/gitlab-com/infrastructure/issues/4410 it was difficult to determine what was going on because of the amount of logging which was rotated off the host quickly.

We should add all of the Prometheus/Thanos, and Grafana hosts to centralized logging. We should also add some troubleshooting steps into the `runbooks` project repository. See https://gitlab.com/gitlab-com/runbooks/blob/master/howto/monitoring-overview.md.

* [x]  Prometheus
* [x]  Thanos components (compact, store, query)
* [x]  Grafana
* [x]  Alertmanager
* [x]  Pushgateway

Story points: 5.0
Migrate the sidekiq-workers dashboard to Prometheus

Currently the sidekiq-workers dashboard relies on InfluxDB. Since we are strategically moving from Influxdb to Prometheus, and our current plans are to leave Influxdb behind in Azure, we need to rebuild this dashboard using metrics from Prometheus.

@smcgivern has pointed out that this is an important dashboard which is frequently used, so it's migration should be done migration~4038689 

**References**

* https://performance.gitlab.net/dashboard/db/sidekiq-workers 
* [issues](https://gitlab.com/groups/gitlab-org/-/issues?search=https://performance.gitlab.net/dashboard/db/sidekiq-workers)
* https://gitlab.com/gitlab-com/migration/issues/291
* https://gitlab.com/gitlab-com/infrastructure/issues/1962

Story points: 2.0
Migrate the rails-controllers dashboard to Prometheus

Currently the rails-controllers dashboard relies on InfluxDB. Since we are strategically moving from Influxdb to Prometheus, and our current plans are to leave Influxdb behind in Azure, we need to rebuild this dashboard using metrics from Prometheus.

@smcgivern has pointed out that this is an important dashboard which is frequently used, so it's migration should be done migration~4038689 

**References**

* https://performance.gitlab.net/dashboard/db/rails-controllers 
* [issues](https://gitlab.com/groups/gitlab-org/-/issues?search=https://performance.gitlab.net/dashboard/db/rails-controllers)
* https://gitlab.com/gitlab-com/migration/issues/291
* https://gitlab.com/gitlab-com/infrastructure/issues/1962

Story points: 2.0
Migrate the grape-endpoints dashboard to Prometheus

Currently the Grape Endpoints dashboard relies on InfluxDB. Since we are strategically moving from Influxdb to Prometheus, and our current plans are to leave Influxdb behind in Azure, we need to rebuild this dashboard using metrics from Prometheus.

@smcgivern has pointed out that this is an important dashboard which is frequently used, so it's migration should be done migration~4038689 

**References**

* https://performance.gitlab.net/dashboard/db/grape-endpoints 
* [issues](https://gitlab.com/groups/gitlab-org/-/issues?search=https://performance.gitlab.net/dashboard/db/grape-endpoints)
* https://gitlab.com/gitlab-com/migration/issues/291
* https://gitlab.com/gitlab-com/infrastructure/issues/1962

Story points: 2.0
Make sure GCP sizing is updated with changes done in Azure week of 6/3

We should go back through the items we had done in https://gitlab.com/gitlab-com/infrastructure/issues/4314 and make sure we scale up GCP hosts similar to what we had done in Azure as needed.

* [ ] API nodes
* [ ] Sidekiq workers
* [ ] Worker chef roles for sidekiq queues
* [ ] Any further tuning of NFS nodes
* [ ] Pgbouncer changes if any
* [ ] Prometheus?

cc @andrewn 

Story points: 1.0
Bastion Access to gprd

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **Bastion** access to `gprd`

- [ ] @meks
- [ ] @rymai
- [ ] @bikebilly
- [ ] @DylanGriffith
- [ ] @fjsanpedro
- [ ] @felipe_artur
- [ ] @mkozono
- [ ] @jprovaznik

How to access Bastions: https://gitlab.com/gitlab-com/runbooks/blob/master/howto/gprd-bastions.md

Story points: 1.0
Bastion Access to gstg

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **Bastion** access to `gstg`

- [ ] @meks
- [ ] @rymai
- [ ] @bikebilly
- [ ] @DylanGriffith
- [ ] @fjsanpedro
- [ ] @felipe_artur
- [ ] @jprovaznik

How to access Bastions: https://gitlab.com/gitlab-com/runbooks/blob/master/howto/gstg-bastions.md

Story points: 1.0
SSH, Shell Access in Production

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **SSH and Shell** access in Production

- [ ] @digitalmoksha
- [ ] @nick.thomas

Story points: 1.0
SSH, Shell Access to Staging

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **SSH and Shell** access on Staging

- [ ] @digitalmoksha

Story points: 1.0
Rails Access to Production

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **GitLab Rails** access on to Production

- [ ] @digitalmoksha


Please note that this also requires https://gitlab.com/gitlab-com/infrastructure/issues/4316 to be completed.

https://gitlab.com/gitlab-com/infrastructure/issues/4316

Story points: 1.0
GitLab Admin Access in Production

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **GitLab Admin** access on https://gitlab.com

- [x] @andrewn
- [x] @digitalmoksha
- [ ] @meks

Story points: 1.0
GitLab Admin Access on Staging

Reference: https://docs.google.com/spreadsheets/d/1O6BrllYss6lnKyguK5dTrjlqsaZyFT0oqOtI_9_e-hE/edit

Reference: https://gitlab.com/gitlab-com/migration/issues/459

In order to conduct the ~"GCP Migration" production and failover rehearsals, please grant the following GitLab team members **GitLab Admin** access on https://staging.gitlab.com

- [x] @andrewn 
- [ ] @meks 
- [x] @rymai 
- [x] @dawsmith

Story points: 1.0
Allow rails-console, db-console and nfs users access to the Bastion hosts

Proposed fix in https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2054 ....

Currently users with `rails-console`, `db-console` and `nfs` access are not allowed onto the bastion hosts, so are not able to access the hosts that they previously could.

It would be good if their previous access was restored.

Story points: 1.0
Remove`ci.gitlab.com` from haproxy

We no longer need ssl configuration for ci.gitlab.com, this can be removed from the haproxy cookbook.


Original description:

According to @jarv, our HAproxy configuration for GitLab.com includes a section for the domain name `ci.gitlab.com`

This has no counterpart in gprd and no entry in the DNS.

Does it require any action for the migration~2977716 or can we assume it's obsolete and remove the HAProxy configuration currently on GitLab.com ?

/cc @jarv

Story points: 1.0
Set up IMAP Prometheus exporter for monitoring Service Desk

Right now we are blind as to whether mail_room is doing anything for Service Desk or reply-by-email.

A quick thing we could do to improve this is to run an IMAP Prometheus Exporter, such as https://github.com/camptocamp/imap-mailbox-exporter.
We'd need to fork it to export the `Unseen` messages in https://godoc.org/github.com/mxk/go-imap/imap#MailboxStatus, but that shouldn't be hard.

At the minimum, I might just run a local exporter and watch it during the GCP failover.

Story points: 1.0
Investigate why postgresql service shutdown on `postgres-01-db-gprd` and did not restart

About four hours ago, postgres-01 went down and did not come back up for some reason:

```
# sudo gitlab-ctl status
run: consul: (pid 1857) 15370s; run: log: (pid 1850) 15370s
run: logrotate: (pid 28186) 969s; run: log: (pid 1863) 15370s
down: postgresql: 15370s; run: log: (pid 1858) 15370s
run: remote-syslog: (pid 1826) 15370s; run: log: (pid 1825) 15370s
down: repmgrd: 1s, normally up, want up; run: log: (pid 1871) 15370s
```

Manually starting again seems to work. I didn't see anything in the log that suggested why it didn't come up automatically.

Story points: 2.0
Sidekiq not running on staging.gitlab.com

Noted by @toon (thanks!)

![Screenshot_from_2018-05-03_14-09-50](/uploads/d00b8f946dcca7340eeb87d5cd1d325c/Screenshot_from_2018-05-03_14-09-50.png)

This blocks any attempt at a planned failover.

Story points: 2.0
Upgrade to haproxy v1.8.x

We are running haproxy 1.6.3 which is goes allll the way back to 2015.
In order to support TLS1.3 and probably some other things we should consider upgrading.
Suggest we role this out on staging and then canary first. I don't believe there will be any problems but @northrup may know better.

Putting in the next next milestone for scheduling consideration.

Story points: 4.0
Access to create a project on our Google API

I'm currently working to deploy a tool that would link our SFDC instance with Google Sheets. This would make it much easier for our reps to mass update their opportunities. 

One of the steps of the installation/setup is to create a custom [Google API Project](https://appexchange.salesforce.com/servlet/servlet.FileDownload?file=00P3A00000XbZCaUAN) - Page 12 of the install instructions. 

I would need access to create a project and edit that project within our instance. I also may have to do this twice (once for an installation and once for a production installation) so I'm not sure if that affects the permission set I would need. I could always update the project once it's created though

Story points: 1.0