giseldo/neodataset huggingface dataset:
Centralize GCP bootstrap module

As a first step to breaking out terraform modules into separate repos, we started using data sources to reference the bootstrap scripts via HTTPS to the files under https://gitlab.com/gitlab-com/gitlab-com-infrastructure/tree/master/scripts/google. This has two issues -- first, the primary source for the bootstrap files used to provision gitlab.com cannot be kept in gitlab.com, so they need to be hosted on ops.gitlab.net; second, due to permissions in the ops instance, they will no longer be available via anonymous HTTP(S) URLs.

1. [x] Create a new bootstrap module that can be included in other modules and provide bootstrap/teardown script content by version
1. [x] Configure push mirroring from ops.gitlab.net -> gitlab.com for the new module
1. [x] Replace `data_source` resources in current modules with references to the new `bootstrap` submodule
1. [x] Clean up bootstrap scripts from `gitlab-com-infrastructure` repository

Story points: 1.0
Terraform Runs are not clean in any environment

(except DR)

This is preventing us from being able to confidently run terraform.  All members of the team should be able to run terraform for a given environment and only see the changes related to the issue they are working on.  Right now there are a wide variety of changes, which leads to members of the team targeting only the work they are doing, and slowly we are building up a list of things that terraform wants to change.  At some point, this is going to bring a bout of destruction that we should avoid.

Utilize this issue to keep track of the work necessary to get all environments into a clean run.

Story points: 5.0
PrometheusUnreachable in DR environment

Now that we can log into the prometheus servers in the DR environment, we should no longer be getting `PrometheusUnreachable` alerts from https://alerts.ops.gitlab.net/ - but that alert is still firing.

We created a silence on it when the servers weren't running, but now that they're running, we should see why we're still seeing the alert.

Story points: 1.0
Access error connecting to monitoring in DR environment

We are getting errors when connecting to:

- https://prometheus-app.dr.gitlab.net
- https://alerts.dr.gitlab.net
- https://prometheus.dr.gitlab.net

![Captura_de_pantalla_2019-01-23_a_la_s__16.55.53](/uploads/f69a94ccb140e561c176ad108861a4b7/Captura_de_pantalla_2019-01-23_a_la_s__16.55.53.png)

Story points: 1.0
Setup AWS service accounts for terraform CI

# Overview

While setting up the CI  pipeline in gitlab-com/gl-infra/gitter-infrastructure!94 we started receiving permissions errors during `tf_plan` jobs. The pipeline had been previously setup to use the packer service account, and permissions were temporarily added to enable access to the remote state resources for planning.  Going forward, we need to configure at least four other service accounts, or two accounts with two roles each:

## Remote-state
1. [ ] Create terraform-remote-state IAM account
1. [ ] Add credentials to 1password
1. [ ] Add CI variables
1. [ ] Update `.gitlab-ci.yml` with credentials for `tf_*_remote_state` jobs
1. [ ] Attach terraform-remote-state IAM policy for `terraform plan` (Copy privileges under `DynamoDB`, `S3`, and `KMS` services from `Packer` policy)
1. [ ] Remove `DynamoDB`, `S3`, & `KMS` privileges from `Packer` IAM policy

## Beta
1. [ ] Create terraform-beta IAM account
1. [ ] Add credentials to 1password
1. [ ] Add CI variables
1. [ ] Update `.gitlab-ci.yml` with `tf_*_beta` jobs and credentials
1. [ ] Create terraform-beta-ro [IAM role](https://www.terraform.io/docs/providers/aws/#assume-role) (read-only, for `terraform plan`), pass role ARN by variable
1. [ ] Create terraform-beta-priv [IAM role](https://www.terraform.io/docs/providers/aws/#assume-role) (admin privs, for `terraform apply`), pass role ARN by variable

## Prod
1. [ ] Setup terraform-prod IAM account
1. [ ] Add credentials to 1password
1. [ ] Add CI variables
1. [ ] Update `.gitlab-ci.yml` with `tf_*_prod` jobs and credentials
1. [ ] Create terraform-prod-ro [IAM role](https://www.terraform.io/docs/providers/aws/#assume-role) (read-only, for `terraform plan`), pass role ARN by variable
1. [ ] Create terraform-prod-priv [IAM role](https://www.terraform.io/docs/providers/aws/#assume-role) (admin privs, for `terraform apply`), pass role ARN by variable

Story points: 3.0
Consider moving chef server to GCP

Chef server is currently running at Digital Ocean. We should consider if it would be better to move it to GCP instead. This issue is for discussing the pros and cons. A followup issue should be opened if decide to move Chef to GCP.

### Summarizing discussion:

#### Pro:
* Network and Permission setup might become easier
* one less one-off service to take care of
* We could upgrade Chef and Ubuntu while keeping the old Chef around as a fallback

#### Con:
* If GCP goes down we need to first standup Chef before we can bootstrap GitLab.com.
  * But installing in a different region and tested backups should mitigate the risk.

## Conclusion

We will move chef to GCP. Tracked via #6128.

**Pre-Requirements are working backup and restore procedures for Chef (#6065, #6075).**

Story points: 3.0
Monitor Chef Erlang Components

We need to find a way to monitor the Erlang-based components of Chef server. There probably is a way to interact with the beam vm to get metrics. We should alert if one of the components is down.

* oc_bifrost
* opscode-chef-mover
* opscode-erchef
* opscode-expander
* opscode-pushy-server
* opscode-solr4

Story points: 5.0
Performance of review apps is slow

Over the last couple of weeks, using review apps has been extremely slow on gitlab.com. The deploy job finishes as expected, but the actual use of the app is slow to the point of being unusable. I have to leave the tab open for several minutes in order to navigate through the app.

I have noticed this on multiple projects, such as https://gitlab.com/gitlab-com/www-gitlab-com and https://gitlab.com/gitlab-org/design.gitlab.com.

Current example that is still up as of 11:30AM PST Jan 22: https://matej-validating-settings-redesign-blog-post.about-src.gitlab.com

slack: https://gitlab.slack.com/archives/C0AR2KW4B/p1548183331596500

Story points: 1.0
Chef client fails on gstg bastion

Looks like `chef-client` fails on `bastion-01-inf-gstg.c.gitlab-staging-1.internal` with:

```
[2019-01-22T17:23:40+00:00] INFO: Client key /etc/chef/client.pem is not present - registering
[2019-01-22T17:23:40+00:00] WARN: Failed to read the private key /etc/chef/validation.pem: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /etc/chef/validation.pem>

================================================================================
Chef encountered an error attempting to create the client "bastion-01-inf-gstg.c.gitlab-staging-1.internal"
================================================================================

Private Key Not Found:
----------------------
Your private key could not be loaded. If the key file exists, ensure that it is
readable by chef-client.

Relevant Config Settings:
-------------------------
validation_key "/etc/chef/validation.pem"

Platform:
---------
x86_64-linux
```

Realized this while figuring out why @cshobe is not yet able to access staging hosts.

Story points: 1.0
Right-size sidekiq nodes

Based on the last 7 days of [metrics](https://prometheus.gprd.gitlab.net/graph?g0.range_input=1w&g0.step_input=86400&g0.expr=avg%20by%20(priority)%20(quantile_over_time(0.95%2C%20instance%3Anode_cpu_utilization%3Aratio%7Btype%3D%22sidekiq%22%2Cpriority!%3D%22utility%22%7D%5B1d%5D))%20*%20100&g0.tab=0&g1.range_input=1w&g1.step_input=86400&g1.expr=avg%20by%20(priority)%20(1%20-%20quantile_over_time(0.01%2C%20instance%3Anode_memory_available%3Aratio%7Btype%3D%22sidekiq%22%2Cpriority!%3D%22utility%22%7D%5B1d%5D))%20*%20100&g1.tab=0) we're grossly over-provisioned for sidekiq server size.

| Priority   | Mem %  | CPU %
|------------|--------|------
| traces     | 22.53% | 4.28%
| asap       | 44.21% | 18.98%
| pages      | 16.22% | 6.26%
| pipeline   | 44.18% | 19.15%
| pullmirror | 35.54% | 16.65%
| besteffort | 50.07% | 40.79%
| import     | 21.45% | 13.47%
| realtime   | 18.92% | 9.02%

We should consider reducing the instance size for these nodes, and possibly adding a few more instances for the queues that need them. We should be able to safely get to 70% utilization on our most active queues without a problem.

This would save us quite a lot of production resources. The napkin math says 50% of an engineer's salary.

Story points: 3.0
Console session logs are missing a "history.*" tag

Extension of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6028, we now have the logs in Stackdriver but we can't easily filter them since they are missing a tag.

I think it is easily fixed by including `tag ${tag}` in the [`record_transformer`](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/blob/master/templates/default/sessions.conf.erb#L28-30) filters.

Story points: 2.0
Registry Fleet | Memory Starvation

From oncall handover meeting on 1/22, we have decided to create an issue for the Registry servers getting rebooted due to memory consumption. (Between the week of 1/15 - 1/22, 14 out of the 19 incidents were due to this). 

The next thing we talked about possibly doing is doing profiling on the registry server(s) since the issues have been happening more and more frequently and see if we can identify the source of the memory leak.

Story points: 2.0
Remove persona authentication from forum.gitlab.com

Persona was an authentication provider by Mozilla, that was discontinued like [2+ years ago](https://developer.mozilla.org/en-US/docs/Archive/Mozilla/Persona)!

I thought we got rid of it, but it seems we still use it on the forum. These are the plugins we should use https://gitlab.com/gitlab-com/runbooks/blob/master/howto/discourse-forum.md#plugins-we-use

## Action

1. [x] Follow steps in https://gitlab.com/gitlab-com/runbooks/blob/master/howto/discourse-forum.md#adding-or-removing-a-plugin and make sure the persona plugin is absent.

Story points: 2.0
Move Chef Repo to Ops Instance

The chef repository is currently mirrored from gitlab.com to the ops instance.  This creates latency between every merge and when it can be applied.  It also means that if GitLab.com is down, we cannot easily push any changes that might be needed to bring it back up.

It is set up like this to optimize for anyone who might want to submit MR's to the repo without having access to the ops instance.  We need to optimize this for the majority of requests.  For the small number of outside requests that we receive for this repository, we can manually apply the patches.

The steps are:
- [ ] Audit and adjust permissions on the chef-repo project in the ops instance
- [ ] Ensure that the repositories are sync'd
- [ ] Change the direction of the sync from the ops instance to the gitlab.com instance
- [ ] Change any CI jobs or other automation to point the chef server to the ops instance
- [ ] Notify all users to change the remotes in their local copies

Story points: 3.0
Develop a framework for automating infrastructure changes

Take https://gitlab.com/gitlab-com/gl-infra/production/issues/633 for example, I want to have a script for it that looks roughly like this:

```ruby
step do
  stop_service 'chef-client', on_role: '{{env}}-base-db-patroni'
end

step do
  pause_patroni_cluster
end

step do
  merge_and_apply 'https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/474'
end

patroni_replica = patroni_replicas.sample
step do
  on_node patroni_replica do
    converge_chef
    restart_service 'consul'
  end

  verify do
    entries = query_consul_dns 'replica.patroni.service.consul.', on: patroni_replica

    should_include patroni_replica.private_ip, in: entries
  end
end

step do
  (patroni_members - [patroni_replica]).each do |patroni_member|
    on_node patroni_member do
      converge_chef
      restart_service 'consul'
    end

    verify do
      role = patroni_member.leader? ? 'master' : 'replica'
      entries = query_consul_dns "#{role}.patroni.service.consul.", on: patroni_member

      should_include patroni_member.private_ip, in: entries
    end
  end
end

step do
  merge_and_apply 'https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/473'
end

step do
  on_all_nodes do
    converge_chef
  end

  verify do
    count = run_in_rails 'puts Gitlab::Database::LoadBalancing.proxy.load_balancer.instance_eval { @host_list }.hosts.map(&:host).count'

    should_equal count, patroni_replicas.count
  end
end
```

Notes about the approach:

* Using a DSL makes the script easy to read and review
* Each action is a Ruby class with separate methods for dry-run and full-run
* Dry-runs should allow checking prerequisites
  * If a step is executing `knife ssh ...`, we can check if knife is properly configured and we can actually SSH into machines
* In full-runs you'll be prompted for step progression (skip, retry, halt, etc...)
* Environment is not mentioned explicitly in the script
* Over time we should grow an inventory of reusable actions
* Logs everywhere
* The framework should not have any gem dependencies

I have a rough structure for the framework, will push the whole thing once we have something that works.

Story points: 5.0
Inventory Catalogue - Version

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Version

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Web

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Web

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - SideKiq

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
SideKiq

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Share

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Share

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Runner

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Runner

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Registry

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Registry

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - PSQL Timing

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
PSQL Timing

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Prometheus App

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Prometheus App

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Prometheus

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Prometheus

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Postgres DR Delayed

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Postgres DR Delayed

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Postgres DR Archive

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Postgres DR Archive

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Postgres

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Postgres

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Patroni

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Patroni

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 1.0
Fix postgres wal archiving on chef.gitlab.com before running out of disk space

WAL logs are piling up in `/var/opt/opscode/postgresql/9.2/data/pg_xlog` since 2018-10-03 because the wal-e archive command is failing. It is taking 250GB on disk already (92% disk usage) so we need to fix it before running out of space.

```
2019-01-18_13:42:35.84557 LOG:  archive command failed with exit code 1
2019-01-18_13:42:35.84559 DETAIL:  The failed archive command was: /usr/bin/envdir /etc/wal-e.d/env /opt/wal-e/bin/wal-e wal-push pg_xlog/00000001000000F3000000F8
2019-01-18_13:42:37.01083 wal_e.main   ERROR    MSG: no storage prefix defined
2019-01-18_13:42:37.01085         HINT: Either set one of the --file-prefix, --gs-prefix, --s3-prefix or --wabs-prefix options or define one of the WALE_FILE_PREFIX, WALE_GS_PREFIX, WALE_S3_PREFIX, WALE_SWIFT_PREFIX or WALE_WABS_PREFIX, environment variables.
```

We need to monitor for successful db backups.
Chef monitoring is tracked in #5951.

Story points: 2.0
Problem statement and defining a goal for "PG Repack" epic

The epic for "PG Repack" is not well defined (there's no problem statement nor a defined goal).

Let's start with defining the problem and setting a goal for the epic. Ultimately this will be the beginning of the "design document" and also become the description of the Epic.

I would like the epic to be specific and measurable, such that we know what we're aiming for (see [SMART](https://en.wikipedia.org/wiki/SMART_criteria)).

Story points: 3.0
Inventory Catalogue - Pages

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Pages

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Mailroom

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Mailroom

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0