giseldo/neodataset huggingface dataset:
Unable to run terraform against GCP when S3 is unavailable

I had a strange issue today where I was working on the DR environment, running terraform commands to configure resources in GCP and for some reason I lost connectivity to AWS and S3 from my location. I could ping Amazon addresses, but could not connect to either S3 or the AWS console. All of my other connectivity to GCP and elsewhere was unaffected.

This should not have been a problem, since I was only working in GCP, but since the `.tfstate` file lives in S3, I could not use terraform at all.

I'd like to propose moving the `.tfstate` files to Google Storage. It would be nice to reduce the dependencies for critical components like this.  If this happened during a high pressure incident, it would be extremely frustrating.

Does anyone know of any reason why we shouldn't do this? 

cc/ @gitlab\-com/gl\-infra

Story points: 1.0
Allocate more storage on about-src.gitlab.com

The `/home` filesystem on `about-src.gitlab.com` filled up over the weekend, preventing users from updating the website. I managed to free up ~600MB of old/unnecessary files, but will need further assistance from someone more familiar with this system on either a) which files under `/home/gitlab-runner` can be deleted to free up space, b) extending a volume in Azure, or c) both.

/cc @gitlab\-com/gl\-infra

Story points: 3.0
license.gitlab.com restarting every 30 minutes

This looks like Chef restarting it for some reason:

```
root@license:/var/log/upstart#  grep "master process ready" license-gitlab-com.log
I, [2018-12-22T07:08:59.159445 #27739]  INFO -- : master process ready
I, [2018-12-22T07:40:41.871684 #29583]  INFO -- : master process ready
I, [2018-12-22T08:15:51.267262 #31442]  INFO -- : master process ready
I, [2018-12-22T08:50:59.267311 #832]  INFO -- : master process ready
I, [2018-12-22T09:21:37.651062 #2663]  INFO -- : master process ready
I, [2018-12-22T09:54:35.334625 #4508]  INFO -- : master process ready
I, [2018-12-22T10:27:24.003285 #6353]  INFO -- : master process ready
I, [2018-12-22T10:58:08.655725 #8164]  INFO -- : master process ready
I, [2018-12-22T11:30:36.299489 #10008]  INFO -- : master process ready
I, [2018-12-22T12:05:06.211216 #11835]  INFO -- : master process ready
I, [2018-12-22T12:40:30.717682 #13675]  INFO -- : master process ready
I, [2018-12-22T13:14:45.559951 #15540]  INFO -- : master process ready
I, [2018-12-22T13:47:28.384103 #17386]  INFO -- : master process ready
I, [2018-12-22T14:20:29.356144 #19244]  INFO -- : master process ready
I, [2018-12-22T14:51:56.199960 #21080]  INFO -- : master process ready
I, [2018-12-22T15:22:52.262106 #22910]  INFO -- : master process ready
I, [2018-12-22T15:56:28.324569 #24759]  INFO -- : master process ready
I, [2018-12-22T16:30:37.367573 #26623]  INFO -- : master process ready
I, [2018-12-22T17:05:39.551101 #28488]  INFO -- : master process ready
I, [2018-12-22T17:36:18.146337 #30317]  INFO -- : master process ready
I, [2018-12-22T18:10:18.243692 #32175]  INFO -- : master process ready
I, [2018-12-22T18:45:39.000223 #1579]  INFO -- : master process ready
I, [2018-12-22T19:18:47.696688 #3447]  INFO -- : master process ready
I, [2018-12-22T19:53:59.259197 #5287]  INFO -- : master process ready
I, [2018-12-22T20:26:07.063813 #7141]  INFO -- : master process ready
I, [2018-12-22T20:58:10.156486 #9312]  INFO -- : master process ready
I, [2018-12-22T21:32:11.070613 #11165]  INFO -- : master process ready
I, [2018-12-22T22:04:36.984267 #13026]  INFO -- : master process ready
I, [2018-12-22T22:36:17.337633 #15085]  INFO -- : master process ready
```

/cc: @ctbarrett

Story points: 2.0
Move terraform modules to the ops instance

[RCA](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5813) corrective action:

> failed due to inaccessible terraform modules not hosted in ops instance

We should move all modules to the ops instance and make them the source of truth for push and pull. Like other repositories on ops.gitlab.net we can setup a push mirror so they are mirrored on gitlab.com.

Story points: 2.0
Beefier GitLab shared runners

In https://gitlab.com/gitlab-com/www-gitlab-com/issues/2342#note_84878793, @rspeicher noted that if we increased the power of our runners for www-gitlab-com builds, we'd save a lot of time for those builds to happen.

The machines have changed since that test, but mostly as part of the GCP switch rather than any dramatic change in specs; @SteveAzz and @tmaczukin confirmed (in https://gitlab.slack.com/archives/C0SFP840G/p1545226526151000?thread_ts=1545226152.150800&cid=C0SFP840G) that these are n1-standard-2.

In https://gitlab.com/gitlab-com/www-gitlab-com/issues/2342#note_126680858, @northrup said:

> We have a pool
that is in GCP for our use that we pay for, it makes no sense to have the
lowest form of box which takes more time because I think the combined
wasted time waiting on CI in employee pay is more than what we’d pay to
just spin up and down higher CPU count boxes.

And I agree! Can we beef these up, please? Even having an eight-core machine just for the website's master pipeline would be great, but if we could extend that across all GitLab projects it would probably help a lot.

Story points: 1.0
sentry.gitlab.net is slow (Sentry error reporting)

https://sentry.gitlab.net/ is being really slow and I can barely load an issue there

I even got this error once,

> Frig
> 
> Something went horribly wrong rendering this page. We use a decent error reporting service so this will probably be fixed soon. Unless our error reporting service is also broken. That would be awkward. Anyway, we apologize for the inconvenience.
>
> `TypeError: Cannot read property 'id' of undefined`

![](https://i.imgur.com/FaVBmr8.png)

---

Are there dashboards for Sentry CPU, etc?

---

 - Sentry error reporting

Story points: 1.0
Registry nodes scale down

Recently we spun up extra nodes due to combat a memory leak, but all nodes are still leaking memory, so we've created a job to daily restart these services.

Since this problem is systematic and not dependent on scale, let's rid of the extra nodes.

References:
* https://ops.gitlab.net/gitlab-com/gl-infra/registry-restarter/pipeline_schedules
* https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/640/diffs

Story points: 1.0
Evaluate our log retention needs

Aggregating logs at a cloud service provider like Elastic is bound to costs and eventual storage/bandwidth capacity limits. To keep costs at a reasonable amount we should do some research:

* what is the bandwidth/storage limit of Elastic?
* What is the current rate of logs send to Elastic? What do we expect in the future?
* which types of logs do we need in Elastic (e.g. do we need debug logs)?
* How long do we need to keep each of the log types?
* Which log types are we storing currently?
* How can we reduce the amount of logs send to Elastic?
* Do we need to look for alternatives to Elastic cloud?

(related to #5757)

Story points: 3.0
Increase Elastic Cloud storage watermarks

Storage is filling up on our elastic cloud. When reaching the storage low-watermark on a node, shards will be moved to another node but if all nodes have reached the low-watermark, the cluster will stop storing any data. As per suggestion from Elastic (https://gitlab.com/gitlab-com/gl-infra/production/issues/616#note_124839760) we should increase the low-watermark to just leave about 150gb free, so we can make better use of storage capacity.

Story points: 2.0
Canary instances are reporting in prometheus as `stage=main`


https://prometheus.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=count(%7Bstage%3D%22main%22%2C%20fqdn%3D~%22.*cny.*%22%7D)%20by%20(job%2C%20fqdn%2Cstage)&g0.tab=1

```
{fqdn="fe-cny-01-lb-gprd.c.gitlab-production.internal"}	3
{fqdn="fe-cny-02-lb-gprd.c.gitlab-production.internal"}	3
{fqdn="fe-cny-01-lb-gprd.c.gitlab-production.internal",job="node"}	471
{fqdn="fe-cny-01-lb-gprd.c.gitlab-production.internal",job="mtail"}	157
{fqdn="fe-cny-01-lb-gprd.c.gitlab-production.internal",job="fluentd"}	13
{fqdn="fe-cny-01-lb-gprd.c.gitlab-production.internal",job="process_exporter"}	84
{fqdn="fe-cny-02-lb-gprd.c.gitlab-production.internal",job="process_exporter"}	84
{fqdn="fe-cny-02-lb-gprd.c.gitlab-production.internal",job="node"}	471
{fqdn="fe-cny-02-lb-gprd.c.gitlab-production.internal",job="mtail"}	224
{fqdn="fe-cny-02-lb-gprd.c.gitlab-production.internal",job="fluentd"}
```

cc @T4cC0re @jarv

Story points: 1.0
Gitter OS Patching (beta)

## Summary
As an interim step towards updating Gitter AMI images (infrastructure#5755), follow-up to infrastructure#5492, and in support of production#620, we need to perform OS patching on all Gitter instances to establish timing and validate the steps listed in production#620.

## Patch process
The patch process should at least be scripted, or possibly implemented via Ansible. The broad strokes will require running `apt-get update && apt-get upgrade`. On at least one instance, preferably one instance of each service/group, we will also need to script and validate the rollback process before proceeding to production. Finally, we need to consider service impact when orchestrating the change, by draining connections and detaching instances from load-balancer(s) wherever possible.

/cc @gitlab\-com/gl\-infra @MadLittleMods

Story points: 1.0
Convert environments to associate branches with deployments instead of tags

The next step for the environments project is to correct the design decision to use tags to track deployments. 

It makes much more sense to track a deployed environment with a branch, and use tags to track versions of the code. The original design was using tags, because they could overload meta information into the tag name, but there are more intuitive - and less cool - ways to get that information, so I’m going to convert it to the boring solution that’s easy to understand, and works the way the rest of the world works.

Story points: 3.0
Change environments project to use single pipeline

There is a design decision I don't like on the original project.  They were working around a limitation in the CI pipelines by running multiple pipelines for each deploy. I can see why they did it, because you can only expand a variable once per pipeline. There are better ways to work around it though. 

I am converting it to use one pipeline per deploy, which means that we can do automated rollbacks.

Story points: 3.0
Rotate secret_key for customers.gitlab.com

Hi team, can you please change the [`secret_token`](https://gitlab.com/gitlab-cookbooks/cookbook-customers-gitlab-com/blob/master/attributes/default.rb#L11) that's stored in the chef vault?

REF: https://gitlab.com/gitlab-org/customers-gitlab-com/issues/379#note_124194313

Story points: 1.0
Database Reviews

* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8637#note_124023965
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8497#note_124026675
* [x] @NikolayS, @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23464#note_124871412
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8871#note_125248109
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8871
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23630#note_125506227
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8939
* [ ] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8927#note_126726752
* [ ] https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8669#note_126158494
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19740#note_125938200
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23098#note_125777941
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/issues/54643#note_125483243

Open reviews have been moved over to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5851.

Story points: 3.0
Upgrade linux kernel in production environment

This issue is to keep track of the planning, reviews and execution (via CR) to upgrade linux kernel from `4.10.0-1009` to `4.15.0-1015` in production environment. 

For past reference the following were completed for staging:
- https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5289
- https://gitlab.com/gitlab-com/gl-infra/production/issues/607

Lessons learned from upgrading the kernel in staging:
- [ ] Revert the kernel on redis, sentinels, postgres, file-*, prometheus, alertmanager in `STAGING`
- [ ] Don't upgrade kernel for redis, sentinels, postgres, file-*, prometheus, alertmanager
- [ ] Include a dry-run functionality which would just print a list of hosts to be upgraded
- [ ] Link script in the production CR (once it is created) so it can be reviewed along the CR itself as well. 

Story points: 8.0
Setup automated AMI builds and OS updates for Gitter

Currently, we configure [unattended upgrades](https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/blob/master/ansible/roles/base/tasks/packages.yml#L98-104) to install [security updates](https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/blob/master/ansible/roles/base/files/etc/apt/apt.conf.d/50unattended-upgrades). This is great for a lower bar, but we also need a process to keep base images updated (packer pipelines) for minimal bootstrapping of new nodes in autoscaling groups, and we need to update to the latest LTS version of ubuntu across the fleet.

1. [ ] Trigger packer builds for each role when Ansible code is updated
1. [ ] Trigger daily packer builds to bake in latest OS package updates (`apt-get update && apt-get upgrade`)
1. [ ] Implement scheduled scaling actions to cycle ASG nodes
1. [ ] Implement cleanup job/lambda for pruning old AMI images

These are just some high-level ideas to prompt a more intensive review/design doc, followed by more specific implementation issues.

Story points: 13.0
Enable Elasticsearch on ops.gitlab.net

I suggest that we enable Elasticsearch on ops.gitlab.net for a number of reasons:

1. We don't test/use Elasticsearch in a real environment. This provides an opportunity to do some basic testing.
2. Global search would be helpful in looking for content across cookbooks.
3. It will expose limitations with Geo support etc.

@vsizov @mdelaossa @jarv Thoughts?

Story points: 1.0
Fix links in Postgres Grafana Dashboards

Since the Patroni migration, the Postgres Dashboards have undergone several adjustments, leading to several links pointing to non-existent Dashboards within Grafana. For example the links from the [Top Qeries Dashboard](https://dashboards.gitlab.net/d/000000242/postgresql-top-queries?orgId=1&var-environment=gprd&var-fqdn=patroni-01-db-gprd.c.gitlab-production.internal&var-datname=gitlabhq_production&var-user=gitlab&var-interval=6h&var-prometheus=prometheus-01-inf-gprd) to the [Query Drilldown Dashboard](https://dashboards.gitlab.net/d/000000237/postgresql-query-drill-down?orgId=1) for each of the top query IDs. They should be fixed.

Story points: 2.0
chatops command for oncall does not show overrides

It looks like when there are overrides for oncall they aren't shown.
![Screen_Shot_2018-12-12_at_3.13.57_PM](/uploads/ee6684e557b8069ee64013fdcdc5d165/Screen_Shot_2018-12-12_at_3.13.57_PM.png)

Story points: 2.0
AWS Rate Limiting

Our access to our AWS project was being rate limited by AWS because of excessive usage of the API / resource calls.

It was to the point that it was even effecting the GUI console throwing "Rate Exceeded" errors.

After investigation it was determined that some automation for `gitlab-review.app` was creating the excessive usage. There were also more than 10,000 DNS records in the `gitlab-review.app` domain.

To remove the throttling without breaking _too_ many things I removed the role policy (`Review-app-ee-dns`) from the user (`review-app-ee-dns`) which removed access.

Story points: 4.0
Consider moving gitlab-production to committed use discounting

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6086, we investigated where we might be able to apply [committed use discounting](https://cloud.google.com/compute/docs/instances/signing-up-committed-use-discounts) to our GCP compute spend. Two ideas emerged:

1. [Apply CUDs to our CI runner fleet](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6238).
1. Move gitlab-production off [sustained use discounting](https://cloud.google.com/compute/docs/sustained-use-discounts) and on to committed use discounting (this issue).

Since committed use discounts (CUDs) offer significantly deeper discounting than sustained use discounts (SUDs), this issue is to investigate the possible savings and action plan for this move.

### Proposal

* Identify the gitlab-production systems and compute we could move to CUDs, even if they're currently benefitting from SUDs.
* Estimate the additional discounting and possible savings. Evaluate whether or not it's worth pursuing based on the risk (we'd be committing to the spend).

### Links/resources

* [CloudPark.com](https://www.parkmycloud.com/blog/google-cloud-committed-use/) on Google CUDs

Story points: 4.0
Chef does not complete due to broken td-agent

`td-agent` is not starting on the DR database servers (and possibly other servers as well).  This is causing chef to fail.

The problem seems to be multiple versions of the `googleauth` gem. Deleting `0.8.0` allows `td-agent` to start, but the next chef run puts it back and it starts failing again.

```
/opt/td-agent/embedded/lib/ruby/site_ruby/2.4.0/rubygems/specification.rb:2280:in `check_version_conflict': can't activate googleauth-0.6.6, already activated googleauth-0.8.0 (Gem::LoadError)
```

This issue is related to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6224 - but not the same issue. They both result in chef not completing a run.  The other one fails during bootstrapping.  This one fails later.

Story points: 3.0
Bump `client-output-buffer-limit` for `redis` nodes

From https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6228:

> Set the hard limit to 4gb as having the same hard and soft limits defies the purpose of the latter.

> Note that `chef-client` doesn't run `gitlab-ctl reconfigure` on `redis` nodes, and we can't run it ourselves as it will restart the service upon changing `redis.conf`. So updating the file manually and running `config set <new config>` in a Redis console is our best approach. **Experiment in staging first!**

Story points: 2.0
Chef Runs broken while bootstrapping

Chef runs are broken on the DR database servers. Machines won't bootstrap.

It appears to be because of this file missing `/opt/prometheus/node_exporter/metrics/chef-client.prom`

The error is:

```
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] ERROR: Running exception handlers
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] ERROR: PrometheusHandler: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /opt/prometheus/node_exporter/metrics/chef-client.prom>
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] ERROR: Exception handlers complete
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
Feb 13 23:12:29 patroni-02-db-dr startup-script: INFO startup-script: [2019-02-13T23:12:29+00:00] ERROR: undefined method `missing_extensions?' for nil:NilClass
```

This has been very difficult to debug since it happens before the users and ssh keys are set up, so `/var/chef/cache/chef-stacktrace.out` is unavailable. Logging in via the GCP console is not working, and the serial port presents with a login prompt which is useless without credentials.

This worked when the machine was initially set up, using the same chef-module versions. Now after re-creating the instance, I can't get any combination of versions to result in a successful initial chef run.

Story points: 3.0
Gather rough estimation of non http traffic for cloudflare estimate

Making an issue to make notes on collection of this estimate:

Thank you for your response. I apologize if my email wasn't clear on the specific Spectrum (non-HTTP/S) questions from the PM.  What he is looking to obtain:
1. How much of the 360TB/mo will be routed through non-standard HTTP/S ports? 
2. How many concurrent SSH connections do you have today? 

Plan for 1:
Get an "average" from queries like:
increase(haproxy_frontend_bytes_out_total{fqdn=~".*altssh.*.lb.*", job=~"haproxy", frontend!="stats"}[24h] offset 1d)

Plan for 2:
haproxy_backend_current_sessions{fqdn=~".*altssh.*.lb.*", job="haproxy", backend!~"stats.*"}

Story points: 1.0
Design Document for the process of restoring Database from GCE disk snapshots

Rather than wait for ZFS database snapshots, and because we're not relying on snapshot quality at the level necessary for disaster recovery or customer facing functionality, we're going to automate the extraction of GCE disk snapshots from the production environment for use in our testing environment(s).

The design should outline the process with technical detail. It is not limited to, but must pay attention to ways we will:

- [ ] protect customer data by treating it as production data
- [ ] scrub data and do not introduce any personally identifiable information (PII) or intellectual property (IP) into the testing environment
- [ ] ensure the proper safeguards are in place, potentially even moving the process of scrubbing data to it's own isolated environment (to avoid accidentally manipulating production data)
- [ ] scale down data for functional use

Story points: 3.0
Monitor packagecloud DB backups

packages.gitlab.com was running out of disk space again because db backups took all the space.
Some of the backups seem to fail and then leave their temporary data on disk (ca. 600 GB for each backup).

We should monitor for backup failures and automatically cleanup data from failed backups.

If that isn't enough we can further trim down the local retention interval (as done in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5820, via https://gitlab.com/gitlab-cookbooks/gitlab-packagecloud/merge_requests/10).

Longterm solution would be to migrate from MySQL to RDS (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5222).

Story points: 3.0
Configure tests for cookbook-license-gitlab-com

As a pre-requisite step for #5817, we need to enable basic `chefspec` and `inspec` test configuration so that we can validate the changes before deploying the updated cookbook.

1. [x] Configure `chefspec` (existing tests)
1. [x] Configure `test-kitchen`
1. [ ] Validate `kitchen converge` (existing/default `inspec` tests)

/cc @sdval

Story points: 3.0
CI/CD handoff - switch alerting to SRE rotation

TBD based on feedback from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6203

The main action will be to change the alerting template for alerts related to ci-cd to not @-mention cicd-ops.
We'll still post to #ci-cd-alerts without the mention.

Second, to follow our existing format - we're going to change the channel to #alerts-ci-cd as all of our other alerts channels are prefixed that way.

cc @ahanselka for noting when we will do those changes.

Story points: 2.0