giseldo/neodataset huggingface dataset:
Setup alerts for both custom and non-custom ssl endpoints for pages

This comes out of the pages postmortem. The front-end for pages ip should never be released. We should have critical (pagerduty) alerts for both staging and production so we are notified immediately if there is an issue. By the time we are paged in production it will be too late so we need to do this for staging as well.

For the alert I think the best thing to do is scrape an ssl custom domain, that is configured to use an A record.


Story points: 2.0
network peer the monitoring subnets in gprd/gstg/ops

For alertmanager clustering it is desired that we network peer the monitoring subnets. This will cover both prometheus and alertmanager.

Note that internal dns resolution will not work across projects so if we do configure cross project connections it will need to be by the private ip address.

For more context see https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/2531#note_134273

- [x] Peer gstg and gprd (see how this was done for the ops network)
- [x] Create firewall rules that allow alertmanagers to communicate with each other

Story points: 2.0
onboarding jfinotto ssh access to the environment

Hello here my .pub 
Username jfinotto

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNidMhz6K0KoKcnM7l08YWTp1T84bjP5tF0u6utfTlpjjqrIFI2mhYicga2nRxypb6xriaiOtR+w9uzVIWdJbePG3mu82vdIRUrk9C/tEQsMy8Y51jUJF/i0JsKMKfTQnrC9cfe0L4zOP/NcQwL0GUtIhc8Nw8T1B/2i7Djz39RYWDEqNlwYMl9NPtjW1067ns+YZw68bKbB5e+ndaheE1BtcmmdU9n9Cx4QB9366GO7UWwx7f9o3Ifd8/NwDsz81rx8ie2rwL4hpbUsDkxwY2DPaZ68MnTScLIdrnekxs/OGmoTBwLf/VZiQ7m3z9tDbQTZMUqBReXVD+354krPD7 jfinotto@Joses-MBP.fritz.box

Story points: 1.0
Check for internal ip addresses when updating our restriction lists

We recently had an incident where the blackbox ip address was flagged and put into a restricted list.

The following two repos:
* https://ops.gitlab.net/gitlab-com/security-tools/front-end-security.git
* https://ops.gitlab.net/gitlab-com/security-tools/recaptcha.git

Would benefit from a pipeline check to ensure that none of the IPs are internally reserved.

Story points: 2.0
[Design Document] Configure properly Autovacuum for postgresql

Please fill with your ideas the following design doc :
https://docs.google.com/document/d/1rQxVHrVb_LGGsG69G7r7-8vTVkN5Slr7eMWw9uxMTFA/edit?usp=sharing

Deliverables:

- [x] [New Document Version](https://docs.google.com/document/d/16uj2mK4k93xumNdU4qWVNmh8EI-yC4e8fw248S5LB-0/edit)
- [x] Change Request. cc/ @gerardo.herzig 

Story points: 12.0
[Design Document] Postgresql Backup & Recovery

Please fill with your ideas the following design doc : 
https://docs.google.com/document/d/1X51du8kpxX4GAmipC2DeT61JDNqTz78sHPgTBkSxa5U/edit?usp=sharing

Once ready, move the design to the Handbook and ping the team to review the MR before this is being merged. Once the MR is merged, the design is considered final.

Handbook for designs: https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/source/handbook/engineering/infrastructure/design/index.html.md

Story points: 0.0
Hurricane Florence Readiness

Our region is us-east1 in Moncks Corner, SC - hurricane path is going to be near there based on NHC maps.

This is a general issue to group together actions we will take to be ready should we see disruptions of service in our regions.

Initial actions:

* [  ] Look into options for alternate region backup of git / nfs data
* [x] Cloud storage is already multiregional - anything else to do?
* [x] Make sure about.gitlab.com and any other important assets are in an alternate region
* [ n/a too big ] Start work in Terraform to build out stack for Geo in us-west1



Story points: 3.0
Services running on the Patroni cluster need to have their limits matching omnibus

gitlab-omnibus runs all of its services with [bumped limits](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/2fc14ae20a136ebb2662383b8d9c97d5f9ff0e1e/config/templates/runit/runsvdir-start.erb#L20-37). We are using custom cookbooks to install pgbouncer and PostgreSQL on the Patroni cluster, so we need to have a way to apply the same limits these services.

Things to take into consideration:
* PostgreSQL is not managed by systemd or the likes, rather, it's spawned by Patroni so we may need to apply the limits to the Patroni service to be inherited by PostgreSQL.
* Bumping such limits will likely need the services to be restarted, which can cause brief disruption of service, so careful planning is needed for production roll-out.

~"corrective action"

Story points: 3.0
pgbouncer mtail doesn't catch some errors

After the Patroni migration, we started seeing this error in the logs: "ERROR accept() failed: Too many open files". It wasn't caught by [mtail regexp](https://gitlab.com/gitlab-cookbooks/gitlab-mtail/blob/b0a57b7bd3b2d18c564d5fccc08081c7673a442d/files/default/mtail/pgbouncer.mtail#L33), so we need to correct it.

~"corrective action"

Story points: 1.0
Setup CI/CD for terraform modules

With each (GCP) module now residing in its own repository, we need to include a few things to move beyond MVC

1. [x] Add a CHANGELOG to each module repo
1. [x] Add a README to each module repo
1. [x] Add a license to each module repo
1. [x] Add a VERSION file to each module repo
1. [ ] Configure CI
    1. [x] Run `tf-lint`
    1. [x] Check `terraform fmt`
    1. [x] Run `terraform validate`
    1. [x] Ensure version gets bumped on merge to master
    1. [x] Automatically tag repo on merge to master

Story points: 6.0
New patcher should post to slack

When running the new patcher CI/CD process, I find myself manually posting a message to slack every time a stage completes.

The automation should do this for me.

At the very least, there should be slack messages for:
- Starting patching staging
- Finished patching staging
- Finished patching canary
- Prompt for manual step to patch production
- Finished patching production

Story points: 2.0
PostgreSQL_ExporterErrors Alerts In Staging

Staging has been sending PostgreSQL_ExporterErrors alerts.  I've been silencing them, but this should really be fixed. It's possible that this alert is no longer necessary and can be deleted, but I don't have enough information to make that call.

```
Labels:
 - alertname = PostgreSQL_ExporterErrors
 - channel = database
 - environment = gstg
 - fqdn = postgres-01-db-gstg.c.gitlab-staging-1.internal
 - instance = postgres-01-db-gstg.c.gitlab-staging-1.internal:9187
 - job = postgres
 - monitor = gstg-default
 - pager = pagerduty
 - provider = gcp
 - region = us-east
 - replica = 01
 - severity = critical
 - stage = main
 - tier = db
 - type = postgres
Annotations:
 - description = This may indicate postgres_exporter is not running or a buggy query in query.yaml on postgres-01-db-gstg.c.gitlab-staging-1.internal
 - title = Postgres exporter is showing errors for the last hour
Source: https://prometheus.gstg.gitlab.net/graph?g0.expr=pg_exporter_last_scrape_error%7Benvironment%3D~%22gprd%7Cgstg%22%7D+%3D%3D+1&g0.tab=1
```

Story points: 1.0
Disk Full on version.gitlab.com due to failing WAL archive - continuation- ensure backups are working

It looks like we need to fix the destination for the archive storage.

```
wal_e.main   ERROR    MSG: no storage prefix defined
        HINT: Either set one of the --file-prefix, --gs-prefix, --s3-prefix or --wabs-prefix options or define one of the WALE_FILE_PREFIX, WALE_GS_PREFIX, WALE_S3_PREFIX, WALE_SWIFT_PREFIX or WALE_WABS_PREFIX, environment variables.
        STRUCTURED: time=2018-12-11T01:14:26.080378-00 pid=23088
2018-12-11 01:14:26 GMT LOG:  archive command failed with exit code 1
2018-12-11 01:14:26 GMT DETAIL:  The failed archive command was: /usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push pg_xlog/000000010000007B0000000D
2018-12-11 01:14:26 GMT WARNING:  archiving transaction log file "000000010000007B0000000D" failed too many times, will try again later
```

Details are in https://gitlab.com/gitlab-com/version-gitlab-com/issues/132

Story points: 1.0
Shared macOS Runners

#### Create Minimal Private Cloud Build for MacOS Shared Runners

Working issue for tracking our macstadium setup and first iteration:

- [x] Sign up for MacStadium account and get information into 1password
- [x] Initiate Minimal Private Cloud Build (1 VPN Firewall, 1 Mac Pro, 1TB Storage, vCenter / vSphere, VMware ESX 6.7)
- [ ] Set up with ESX and first clean base image of mac os x mojave
- [ ] Communicate with CI/CD team an plan next steps for how to setup shared runner manager and shared runners with labels.
- [ ] Configure Mac runners for only tagged jobs, and look for jobs with `mac` tag

This direction lets us make API calls from our platform to vCenter to manage the creation, destruction, and templating of images.

Story points: 3.0
Add https/SSL to next.gitter.im (next feature toggles)

Add https/SSL to [`next.gitter.im`](http://next.gitter.im/) (next feature toggles)

https://github.com/gitterHQ/next.gitter.im

---

Follow-up from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4664#note_123807024

cc @skarbek

Story points: 1.0
Use HTTP data source for bootstrap/teardown scripts

Access to scripts via relative paths produces the following errors on plan:

```
Error: module.sentry.google_compute_instance.instance_with_attached_disk: 1 error(s) occurred:

* module.sentry.google_compute_instance.instance_with_attached_disk: file: open /Users/craig/src/gitlab/gitlab-com/gitlab-com-infrastructure/environments/ops/.terraform/modules/5ddd9c31c4df35aeb0460a824e50024e/../../../scripts/google/teardown-v1.sh: no such file or directory in:

${file("${path.module}/../../../scripts/google/teardown-v1.sh")}
```

This also prevents splitting out modules into standalone repositories, and all these relative `path.module` references should be updated to use the [http data source](https://www.terraform.io/docs/providers/http/data_source.html)

Story points: 1.0
Add ARInsights SPF and DKIM records to update GitLab DMARC authentication policy

# Summary
GitLab is a subscriber to ARInsights software, which is a tool used to monitor analyst relations, typically used by most of the large tech vendors (except MS) and more of the smaller ones. Specifically I use it to send a bimonthly newsletter to industry analysts. We use their SW because it satisfies GDPR requirements, and it also gives me tools to track who opens the newletter and what links they click through on.

This month the newsletter bounced from 5 recipients with a blocked (550) 550 5.7.1 error. All 4 at Redmonk bounced, as well as one at Forrester. When I brought this to ARInsight's attention they wrote me the following:

"Your email administrator needs to add our SPF and DKIM records to update your DMARC authentication policy.  This will enable us to send emails from ARchitect that look like they are coming directly from your domain and will not get trapped in any spam filters. I'm going to have the DMARC records generated and will forward to you with some documentation. 

As soon as the records are generated I'll forward on to you, please don't hesitate to ask any questions as we move through the process."

I will send the files when I receive them. Please let me know what else I need to do.

Story points: 2.0
Our hosted Elastic Search cluster is starting to run low on space

We've been receiving alerts the past week that we start to run out of space, then our cleanup script will come by and clean up some indexes, and we'll receive an alert that we are good to go.  That cleanup script is now reporting 85% used storage space after it completes the cleanup process.

We just scaled up a number of nodes, so log volume has increased recently.  We also have issues (not yet pulled) to start adding a few items into elastic search.  

Utilize this issue and bump up the amount of storage we have available to us such that we don't run out of space in our cluster.  Currently each one of our nodes is setup if 2TB of space, we are using between 1.7TB and 1.8TB on all of them.

Story points: 1.0
Can we redirect all www.about.gitlab.com URLs?

Earlier today we came across an issue where some absolute links caused Google Bot to index and serve pages from www.about.gitlab.com in search results. 

The absolute links have been updated and will eventually move themselves out of Google's Index. To resolve the problem in the interim, can we setup a redirect for www.about.gitlab.com to about.gitlab.com?

Thank you!

Story points: 1.0
Separate repositories for continuous delivery target environments

1. [ ] Create new repository for each environment we currently have new gitlab project
1. [ ] lock old repository to prevent merge-requests/changes while re-organizing repositories (set a push restriction in the gitlab project settings)
1. [ ] Enable/validate pipeline in new repository
1. [ ] Remove environment directory from old repository

## Testing
1. [ ] Copy terraform files from old repository
1. [x] ~~Setup pipeline using image from [`terraform-ci` repo](infrastructure#5689) (n/a)~~
1. [ ] Validate `tf-init` successfully configures remote state & downloads providers/modules
1. [ ] Validate `tf plan` success & output from local repository
1. [ ] Add manual `tf apply` stage
1. [ ] Enable pipeline and validate end-to-end with no-op change

Story points: 5.0
Shared tooling and docker build pipeline for terraform-ci

Create new `terraform-ci` repository for shared tooling/docker images used in each environments' pipeline `gitlab-ci.yml`

This repository will provide a single place to build/manage the container used in all terraform CI/CD pipelines, with required packages and utilities included. The existing terraform wrapper scripts under `/bin` in the `gitlab-com-infrastructure` repository need to be migrated here.

Story points: 2.0
Separate terraform modules repositories

1. [x] Create new `terraform-modules/*` repositories
1. [x] Update all `source = ...` lines in current (sub-)modules to [reference new modules repo/paths](https://www.terraform.io/docs/modules/sources.html#generic-git-repository)
1. [x] Update all `source = ...` lines in current repository to [reference new modules repo/paths](https://www.terraform.io/docs/modules/sources.html#generic-git-repository)

## Testing
From within `gitlab-com-infrastructure`
1. [x] Rename `modules` directory repository to ensure no relative path references remain
1. [x] Remove all `.terraform` directories under `./environments`
1. [x] Execute `./bin/tf-init` for each environment
1. [x] Execute `./bin/tf plan` for each environment
1. [x] Remove renamed `modules` directory

Story points: 3.0
Service discovery for database load balancing

We support service discovery for database load balancing but apparently we're not using it (I looked at a `database.yml` in production, it seems this is not configured).

Service discovery was implemented with https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5883/diffs by @yorickpeterse .

(I'm not sure yet where this issue is going - but I'm inclined to think that we need service discovery to support flawless failovers.)

Story points: 3.0
[Design Doc] GCP Maintenance Automation

We need to upgrade the kernel version of our GCE VMs. While the process is pretty straight-forward and we completed the task on a staging fleet (`api`) via a change request successfully, there were some areas where we could leverage automation because we are looking at nearly 100 VMs for Staging environment alone and more in Production. In addition, this is a type of activity we could do in the near future as well. 

Here is the design doc for this automation effort, which includes MRs for a proposed project as well. https://docs.google.com/document/d/1u-LtKdu0uSt16IBjsq4szCXq0bCA87Sf0or-Blu55SQ 

Specific asks from the team, by **12/7/18** (if possible):
- Please review the design doc
- Please review the MRs (listed in the design doc) 
- Provide feedback 

Related issues:
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5289
- https://gitlab.com/gitlab-com/gl-infra/production/issues/583

Story points: 5.0
Network Security Diagram for security/customer consumption

On the [Infra production architecture page](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/), we have a current architecture diagram.  To help with security questionnaires, a network security diagram would be good to produce.  This would show VPCs (help answer how we separate production traffic from staging, etc) and then the other network separation we have in place. 

cc @MFarber

Story points: 2.0
Add troupe (Gitter) AWS to offboarding

We need to make sure to remove the user on the `troupe` (Gitter) AWS space when offboarding. We should also remove their SSH key https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure#ssh-to-boxes

I don't have access to the project or it doesn't exist anymore, https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/offboarding.md

https://gitlab.com/gitlab-com/people-ops/employment/blob/b4b4f283bcf47d109a24c2427428b0c29d622375/.gitlab/issue_templates/offboarding.md#L122
```
1. [ ] For former Developers (those who had access to part of the infrastructure), and Production GitLabbers: copy offboarding process from [infrastructure](https://dev.gitlab.org/cookbooks/chef-repo/blob/master/doc/offboarding.md) for offboarding action.
```

---

Spawned from @skarbek finding an old user in the list, https://gitlab.slack.com/archives/C3W3PSR88/p1543863899004000

Story points: 1.0
Database Reviews

* [x] @NikolayS -> @abrandl Recursive CTE https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22308#note_115367888 :eyes: @NikolayS (checked CTE, passed to @abrandl)
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23268
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8603
* [x] Douwe -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23353#note_120477127
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7884#note_120724759 :eyes: @NikolayS (tentatively approved; @abrandl see the comment there)
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23412#note_120700982 :eyes: @NikolayS (already reviewed and even merged; double-checked, looks ok)
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23147#note_120549262 :eyes: @NikolayS (commented, waiting for feedback)
* [x] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8641#note_121316786
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23445
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23098
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550
* [x] @abrandl urgent - missed deliverable: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6878/diffs#note_121418853
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8497#note_121843202
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8603#note_121851696
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23508#note_121871425 :+1: @NikolayS (ready to be approved @abrandl, one minor comment left)
* [x] @abrandl urgent - blocker https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23353#note_121895753
* [x] Quick https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23512/diffs#note_121901715
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6878
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8603
* [x] @NikolayS -> @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23564#note_122262530 :eyes: @NikolayS  (checked, passed to @abrandl)
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23436#note_122432169
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8695#note_122348606
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7885#note_122342732
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23217
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8442#note_122477084
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550#note_122275825
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23599
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23609

Story points: 8.0
Structured logging for docker registry

registry.gitlab.com logs (from /var/log/gitlab/registry/) are not sent as structured logs which makes it hard to find and parse them.
Adjust [registry.conf.erb](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/blob/master/templates/default/registry.conf.erb) to parse fields correctly. Make sure that multiline panic stack traces also are forwarded (see https://gitlab.com/gitlab-org/gitlab-ce/issues/54703 for an example log).

Story points: 3.0
Public Dashboard for GitLab.com is not loading graphs and data

The [GitLab Triage Dashboard](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s&orgId=1) does not load the graphs and or data points. This has been confirmed my multiple people as not working:

![image](/uploads/9d8508ab7bf9bfcf4917324b5f08ffb9/image.png)

Story points: 1.0
Deprecate TLS 1.0 and TLS 1.1 on staging

Like what we have set on canary we think we are ready to apply the TLS update on staging.

## references

- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5152#note_108864376
- https://gitlab.com/gitlab-org/gitlab-ee/issues/7794#note_120941430

Story points: 2.0