giseldo/neodataset huggingface dataset:
Camo proxy security review of design

Hello @gitlab-com/gl-security/appsec 

Could you please review the design work on this epic, particularly:
#6834, #6836, #6907 

#6839 might also be interesting background info.

I *think* we've covered off all the big concerns with security, but would appreciate any input you might have.  If it all looks ok, let us know so we can proceed with implementation.

Story points: 1.0
Camo proxy whitelist configuration and control

On the implementation issue, there is discussion about whitelisting object storage.

Determine:
* What this exactly means and what values it might need
* If there's anything else that might need whitelisting
* How we should manage these values

NB: this is not strictly camo proxy config; it's a GitLab application setting which whitelists domains from being passed through camo proxy

Story points: 3.0
Choose camo implementation

There are three options (at least):

1. Original (https://github.com/atmos/camo)
1. cactus/go-camo (https://github.com/cactus/go-camo)
1. arachnys/go-camo (https://github.com/arachnys/go-camo - fork of cactus with more features)

Ensure there aren't any others that need to be on the list, then do a quick evaluation of each for such criteria as features, deployability, maintenance, etc.  Does not require running or active testing; this can be considered a paper-based exercise.

Story points: 1.0
Organize Project Folders in GCP

Currently, there doesn't appear to be any obvious organization scheme behind the project folders in GCP.  Some top level folders contain only one project, and some nested folders have a lot in them.  There is no obvious schema for why things are where they are. I'll update this diagram as things become more clear.

I'm especially interesting in any history behind any of these locations.


```
gitlab.com
├── Analytics 
│   ├── gitlab-analysis
│   └── karu-gitlab-analytics
├── Customer Success 
│   └── (Empty)
├── Development 
│   └── Frontend
│       └── (1 project)
├── Distribution 
│   └── (4 projects)
├── Infrastructure
│   ├── Environments
│   │   └── (Primary Environments, prod, staging, pre, DR)
│   ├── Ephemeral
│   │   └── (Review apps for environments, temporary projects)
│   ├── Security Products
│   │   └── (2 Gemnasium Projects)
│   └── (10 projects, ops, POC's, tools, review apps)
├── IT Operations 
│   └── (Empty)
├── Marketing
│   └── GitLab-public
├── Migration Testing
│   └── (2 Projects - obsolete?)
├── Monitoring
│   └── monitoring-development
├── Sandbox
│   └── (User sandboxes, demo's, labs)
├── Security
│   └── (2 projects)
├── system-gsuite
│   └── apps-script
│       └── (2 projects)
└── (5 projects with no location)
```

![Screen_Shot_2019-06-03_at_3.01.02_PM](/uploads/49e49f9cad555f08b1813d07d75f6975/Screen_Shot_2019-06-03_at_3.01.02_PM.png)

cc/ @gitlab-com/gl-infra 

Story points: 3.0
k8s-workloads IAM service account needs higher level permissions on clusters

As noted in this job, https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/517681 the IAM service account k8s-workloads needs the ability to create Cluster Role bindings.  We should create the necessary role/role binding for this user and restrict it to the namespace he'll operate on.  Currently that is ONLY the `gitlab` namespace.

/cc @gitlab-org/delivery 
/cc @gitlab-com/gl-infra

Story points: 1.0
Split terraform environments

Our current repository structure for terraform leverages symlinks to reference shared code from multiple environments. This is great for keeping our code DRY, but there are frequently occasions when we need to deploy code to individual environments with follow-on deployments to other environments at some point in the future (testing new changes in staging, for example). With all environments referencing the same files, there is no way to make those changes in a single environment without requiring the same changes be applied to all other environments, holding up deployments to all other environments, requiring only targeted applies (which brings a host of other issues), or allowing configurations to drift over time.

As an initial step towards resolving this and enabling further work toward automated deployments, we will duplicate the current `main.tf` file in each environment referencing the shared assets via symlink, then remove the symlinks. Likewise, any shared `variables.tf` content will be copied/merged into each environment's `variables.tf` file, and the symlinks removed/cleaned up.

Story points: 1.0
Inventory Catalogue - File

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
File

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - ELK

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
ELK

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Contributors

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Contributors

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Consul

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Consul

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Console

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Console

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - Blackbox

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
Blackbox

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Inventory Catalogue - API

## Task 
Fill out the gSIC (GitLab Service Inventory Catalogue) template for the service. The task of filling it out is really up to us as we need to reach out to various team members in order to capture the information. 

## Service
API

## Sample Template Location
https://gitlab.com/gitlab-com/www-gitlab-com/blob/5f6a07f582be3c2a15914f4773b8de8e13a6d7e9/source/handbook/engineering/infrastructure/design/sample/201901_ServiceInventoryCatalogue/service-catalogue.yml

## Questions or Comments about the template?
Reach out to @aamarsanaa

Story points: 2.0
Setup network-level access to database replica for ELT loads

This is the second part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5847: Provide network-level access (VPC + firewall rules) to the archive replica in gprd for the ELT job runner. See the discussion over in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5847.

Details:
* Network access to the archive replica `postgres-dr-archive-01-db-gprd.c.gitlab-production.internal`
* From the `gitlab-analysis` project's network
* Limit access to resources in `gitlab-production` to only said replica

Story points: 2.0
SSLMate Action Required cert renewals - Jan 2019

We have action required emails for:

Check that we need the new certs and renew if so:

* [x] ~~http://sentry.gitlap.com/ - Exp Jan 20~~
* [x] ~~http://runners-cache-3.gitlab.com/ Jan 17~~
* [x] ~~http://runners-cache-4.gitlab.com/ Jan 30~~
* [x] http://forum.gitlab.com/ Feb 13
* [x] http://ee.gitlab.com/ - Jan 26
* [x] http://ce.gitlab.com/ - Jan 26
* [x] http://jobs.gitlab.com/ - Jan 26
* [x] ~~http://alerts.gitlab.com/ - Jan 21~~
* [x] http://prometheus.gitlab.com/ - Jan 21

Story points: 3.0
Many nodes in chef server which are not connecting

There are many nodes in the chef server which are not connecting to the server and pulling data.

These servers should either be removed from the list of known hosts, or should be made to connect and pull data.

I am putting the list here so we can have some discussion first, in case any of them are in this state on purpose.

Here is the list:

```
jjn@thor ~/Workspace $ knife status --hide-by-mins 600
11964 hours ago, lfs1.single.gitlab.com, ubuntu 16.04.
10475 hours ago, sync-nfs-02.geo.gitlab.com, ubuntu 16.04.
10475 hours ago, sync-nfs-01.geo.gitlab.com, ubuntu 16.04.
10263 hours ago, redis-cache-01.db.prometheus-testbed.helm-charts-win, ubuntu 16.04.
10263 hours ago, redis-cache-02.db.prometheus-testbed.helm-charts-win, ubuntu 16.04.
7881 hours ago, omnibus-builder-runners-manager.gitlab.org, ubuntu 16.04.
7770 hours ago, sentry-infra.gitlap.com, ubuntu 16.04.
4449 hours ago, consul-03.inf.stg.gitlab.net, ubuntu 16.04.
4449 hours ago, consul-02.inf.stg.gitlab.net, ubuntu 16.04.
4448 hours ago, consul-01.inf.stg.gitlab.net, ubuntu 16.04.
4116 hours ago, deploy.gitlab.com, ubuntu 16.04.
2569 hours ago, customers.stg.gitlab.com, ubuntu 16.04.
2218 hours ago, contributors-01-sv-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
2217 hours ago, contributors-01-sv-gprd.c.gitlab-production.internal, ubuntu 16.04.
1715 hours ago, contributors.gitlab.com, ubuntu 14.04.
1014 hours ago, blackbox.gitlab.com, ubuntu 16.04.
jjn@thor ~/Workspace $
```

Story points: 1.0
about-src.gitlab.com not downloading latest cookbooks

about-src.gitlab.com is not downloading the latest cookbooks pinned on the chef server.

Since the node `about.gitlab.com` was renamed in DNS to `about-src.gitlab.com` for our CDN SSL configuration, I think the hostname and node name on the Chef server also need to be updated, following the process in https://gitlab.com/gitlab-com/runbooks/blob/master/howto/rename-nodes.md.

/cc @northrup for a sanity check

Story points: 1.0
Chef Service Stale on gprd Patroni Servers

Test by running:
```
$ knife status --hide-by-mins 60
```
Nodes effected are: 

```
8 hours ago, patroni-02-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, pgbouncer-01-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, pgbouncer-03-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, patroni-04-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, patroni-01-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, patroni-05-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, pgbouncer-02-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, patroni-06-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
6 hours ago, patroni-03-db-gstg.c.gitlab-staging-1.internal, ubuntu 16.04.
5 hours ago, patroni-03-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, patroni-02-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, pgbouncer-03-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, pgbouncer-01-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, patroni-04-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, patroni-01-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, patroni-05-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, patroni-06-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
5 hours ago, pgbouncer-02-db-gprd.c.gitlab-production.internal, ubuntu 16.04.
4 hours ago, runner-release-01-inf-ops.c.gitlab-ops.internal, ubuntu 16.04.
```

Story points: 1.0
Stale Chef Runs in Production

Chef runs were stale on several production servers.

```
prometheus-01.nyc1.do.gitlab-runners.gitlab.net

prometheus-01-inf-gprd.c.gitlab-production.internal
prometheus-02-inf-gprd.c.gitlab-production.internal

deploy-01-sv-gprd.c.gitlab-production.internal

file-04-stor-gprd.c.gitlab-production.internal
file-18-stor-gprd.c.gitlab-production.internal
```

Story points: 1.0
Share learnings about Terraform changes from Dec 21 RCA

Part of the discussion on the RCA from the December 21st incident was to document ideas on how to make sure we don't have conflicts in Terraform that could force us to go to the GCP console to clean up things.  
[RCA for Dec 21](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5813)

https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/merge_requests/671 tracks proposed notes about process.  Creating this issue to point discussion to that MR.

Story points: 2.0
Review Gitter security group rules & assignments

As a follow-up to #5492, we need to perform a detailed review of Gitter network security rules, both VPC ACLs and per-instance rules in EC2/VPC Security Groups.

This issue is to track discussion around reviewing EC2/VPC security groups, rules, and assignment to EC2 instances for necessary ingress/egress filtering, and relating any MRs to implement changes.

Story points: 2.0
Database Reviews


Carried over from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5851
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/issues/51854#note_127186345
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8988#note_128739113
* [ ] Fixtures https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8240#note_129069979
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23464#note_126871390
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/issues/54643#note_125483243

New:
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8949#note_131363655
* [ ] https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2734#note_145286
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8949
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24325
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2694
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24368/#cd94b16b88eadd9a4a80a04eefd02fa71e712d63
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24368
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24144#note_132337799
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23596#note_132697035
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9110#note_132657852 << :green\_apple: @abrandl ready for approval
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24198#note_132859150 << :green\_apple: @abrandl ready for approval
* [ ] Add a composite primary key for diff files https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24496
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlabhq/merge_requests/2734
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9281
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9182#note_134446447 << :green\_apple: @abrandl ready for approval
* [ ] int4→int8 https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24512

Forgotten?
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19740 – not yet approved

Story points: 5.0
New GCP Project for GitLab QA GKE Clusters (and maybe more for teams)

We have multiple uses for creating Kubernetes clusters in GKE right now. Most people in the company (including our automated tests) are using the `gitlab-internal` project in GCP for doing this and as the company grows we are reaching our GCP limits quite regularly (most challenging to resolve is the IP range limit for multiple clusters  https://gitlab.slack.com/archives/CB07X8AQ3/p1543741063017000?thread_ts=1543741035.016900&cid=CB07X8AQ3)

We (the Configure team and others) use this for testing GitLab's Kubernetes integration locally and on gitlab.com.

We also create these clusters during automated QA tests. This regularly results in failures due to exhausted resources (eg. https://gitlab.com/gitlab-org/quality/nightly/issues/50 ) and this can break in Master and requires us to ask around for people to clean up after themselves (eg. https://gitlab.slack.com/archives/C02PF508L/p1547228126465300 ) which is all a manual process.

There may be other ways to solve these quota (or IP range exhaustion) problems. Perhaps increasing GCP quotas will help but from what I understand we may need to reconfigure something in GCP to allow more IP addresses to fit in the range (see https://gitlab.com/gl-retrospectives/configure/issues/4#note_127820216) .

Another challenge with having everyone in the company using the same GCP instance is that it's difficult to encourage good habits about cleaning things up since not everybody realises that not cleaning things up could actually be creating blockers for other teams or even breaking our master build. If we had separate projects for QA then at least nobody could break the build but still the Configure team might end up blocked if we run out on `gitlab-internal`. Configure team relies heavily on this to do their work so quite possibly they will need their own GCP project as well.

Story points: 1.0
version goes down like every weekend

For the past couple of weekends, version.gitlab.com goes offline for a bit.  We think we've reached enough installations that enough customers ping the app on the weekend to bring it down.

Currently the server sits using nearly all available RAM.  There's no swap available on this machine.  This is a single instance full stack machine.  The current unicorn worker count is set to 3.  What can we do **SHORT TERM** to alleviate the problem?

Sidekiq, unicorn, and chef are the top memory use offenders.  I'm afraid if we try to bump the amount of workers, we'll run the system out of memory.

References:
* https://gitlab.pagerduty.com/incidents/PC6S6FB
* https://gitlab.pagerduty.com/incidents/PP1FIQS
* https://gitlab.pagerduty.com/incidents/P2OQHHD
* https://gitlab.pagerduty.com/incidents/PFQPBXA

Story points: 1.0
Discussion: Programming/Scripting languages we accept as a team

## Objective
The objective of this issue is to derive a discussion around the programming/scripting languages that we accept (or would want to accept) for usage within our team. This came up from a feedback on a recent automation work which was done in Python and a proposal was suggested to do everything in Ruby going forward. Hence, I wanted to create this issue and gather more feedback to understand the reasoning behind this and whether 'Ruby/Go-only' is ultimately where we, as a team, go. 

## Datapoints 
Starting with the most obvious:
- Our product is a Ruby-on-Rails product [1], [2] 
- The SRE role job description does list "Ruby and/or Go" under a "May be a fit" [3]
- We already have ruby/go scripts that help with our infrastructure work

## Discussion 
Is the answer to this inquiry is as simple as: "We should just go with Ruby/Go in all of our infrastructure related work"? While the above data points are already present, I know we do use bash scripts though it is not specifically called out. It is possible that it is an inherent requirement that we all should know and should be able to use. However, going further than this, what do we think about adopting/using other languages such as Python or even Java (not as a replacement of Ruby/Go but as an addition as long as they get our jobs done)?

We might want to avoid getting into too much in-depth discussion on comparing one language to another (underlying implementation, learning-curve, syntax, performance, community support and adoption, functionalities and features...etc) but I think it might be better if we focus on what do we as a team want and why one language is a better choice over another based on our experience and infrastructure need. 

Some indicators on why I think we could possibly add Python to our list:
- We are on GCP and Google Cloud has an officially supported, Python library (just like Ruby and Go) [4]
- Kubernetes is one of the items on our radar and it has an officially supported Python library similar to Go. But, Ruby is listed on the community supported library. [5]
- If we end up using Ansible, it is also written in Python. 
- Within GCP, if we want to write a Google Cloud Function - it currently only supports Node.JS and Python. 

Note, the above don't talk about one language's performance, syntax, configuration, implementation, features...etc. 

I am not sure what the degree of Ruby/Go experience is for the rest of the team. But as for transparency, my personal experience with Ruby/Go is much less than Python/Java. Therefore, a decision to stick with the former group of languages would mean I would have to ramp up on them as well. However, I will be happy to do so if we do end up sticking to Ruby/Go only. :) 

@gitlab\-com/gl\-infra 

## References
[1] https://gitlab.com/gitlab-org/gitlab-ce
[2] https://gitlab.com/gitlab-org/gitlab-ee
[3] https://about.gitlab.com/job-families/engineering/site-reliability-engineer/ 
[4] https://cloud.google.com/apis/docs/cloud-client-libraries

Story points: 3.0
Enable fast_destroy_uploads flag in production

There is a new feature - async deletion of uploads (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20977), this is behind feature flag and disabled. To make sure this feature works properly, I'd like to do following steps on production:

* run `rake gitlab:cleanup:remote_upload_files` on prod to get current number of orphaned files
* enable `fast_destroy_uploads` feature and wait for 1-2 days
* run `rake gitlab:cleanup:remote_upload_files` and compare number of orphaned files - if there is significant increase of orhpaned files, it might be sign that uploads deletion doesn't work as expected

Running this rake task should not cause dramatic IO load on prod server (as most of checking should be done in object store) and it's expected to run probably few hours - https://gitlab.slack.com/archives/C101F3796/p1547136320445300

I set due date to 21st because based on slack discussion that week would be preferred to do this instead of doing it now

/cc @skarbek @ahanselka

Story points: 1.0