giseldo/neodataset huggingface dataset:
CI/CD Readiness Review Part 3 - DB, backup & restore, Security

Part 3 of CI/CD to SRE 

## Database

- [x] If we use a database, is the data structure verified and vetted by the database team?
- [x] Do we have an approximate growth rate of the stored data (for capacity planning)?
- [x] Can we age data and delete data of a certain age?

## Security

- [ ] Were the [gitlab security development guidelines](https://about.gitlab.com/security/#gitlab-development-guidelines) followed for this feature?
- [ ] If this feature requires new infrastructure, will it be updated regularly with OS updates?
- [ ] Has effort been made to obscure or elide sensitive customer data in logging?
- [ ] Is any potentially sensitive user-provided data persisted? If so is this data encrypted at rest?

## Backup and Restore

- [x] Outside of existing backups, are there any other customer data that needs to be backed up for this product feature?
- [x] Are backups monitored?
- [x] Was a restore from backup tested?


Existing docs to use as reference:
- Notes from recent summary meeting - https://docs.google.com/document/d/1diT_Dt8oE2kls09WZg6YU3twuQMM6u6NmseCsB-IlXg/edit
- Diagrams, etc - https://docs.google.com/document/d/1WYmN5oukY3DK2hPFLPkxwnuyfxES8nNPeDLMTN_KhVM/edit

Acceptance Criteria:
- [ ] Perform the Summary and architecture analysis above.  Gather notes and link to the service catalog:  https://gitlab.com/gitlab-com/runbooks/blob/master/services/service-catalog.yml

Child issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6199

Story points: 2.0
CI/CD Readiness Review Part 2 - Operational Risk Assessment

Part 2 of CI/CD SRE handoff

## Operational Risk Assessment

- [ ] What are the potential scalability or performance issues that may result with this change?
- [ ] List the external and internal dependencies to the application (ex: redis, postgres, etc) for this feature and how the it will be impacted by a failure of that dependency.
- [ ] Were there any features cut or compromises made to make the feature launch?
- [ ] List the top three operational risks when this feature goes live.
- [ ] What are a few operational concerns that will not be present at launch, but may be a concern later?
- [ ] Can the new product feature be safely rolled back once it is live, can it be disabled using a feature flag?
- [ ] Document every way the customer will interact with this new feature and how customers will be impacted by a failure of each interaction.
- [ ] As a thought experiment, think of worst-case failure scenarios for this product feature, how can the blast-radius of the failure be isolated?

Existing docs to use as reference:
* Notes from recent summary meeting - https://docs.google.com/document/d/1diT_Dt8oE2kls09WZg6YU3twuQMM6u6NmseCsB-IlXg/edit 
* Diagrams, etc - https://docs.google.com/document/d/1WYmN5oukY3DK2hPFLPkxwnuyfxES8nNPeDLMTN_KhVM/edit

Acceptance Criteria:
- [ ] Perform the risk assessment above.  Gather notes and add to the service catalog:  https://gitlab.com/gitlab-com/runbooks/blob/master/services/service-catalog.yml

Child issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6199

Story points: 2.0
CI/CD Readiness Review Part 1 - Summary and Architecture

Part 1 of the CI/CD Operational work.

## Summary

- [ ] Provide a high level summary of the CI/CD feature sets
- [ ] What metrics, including business metrics, should be monitored to ensure will this feature launch will be a success?

## Architecture

- [ ] Add architecture diagrams to this issue of feature components and how they interact with existing GitLab components. Include internal dependencies, ports, security policies, etc.
- [ ] For each component and dependency, what is the blast radius of failures? Is there anything in the feature design that will reduce this risk?
- [ ] Where applicable, explain how we scale and any potential single points of failure in the design.

Existing docs to use as reference:
* Notes from recent summary meeting - https://docs.google.com/document/d/1diT_Dt8oE2kls09WZg6YU3twuQMM6u6NmseCsB-IlXg/edit 
* Diagrams, etc - https://docs.google.com/document/d/1WYmN5oukY3DK2hPFLPkxwnuyfxES8nNPeDLMTN_KhVM/edit

Acceptance Criteria:
- [ ] Perform the Summary and architecture analysis above.  Gather notes and link to the service catalog:  https://gitlab.com/gitlab-com/runbooks/blob/master/services/service-catalog.yml

Child issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6199

Story points: 6.0
Database Reviews

* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24743
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24986#note_139344877 along with https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9472
* [x] @abrandl https://dev.gitlab.org/gitlab/gitlab-ee/merge_requests/776#note_151375
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22743
* [ ] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9445
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25299#note_142327479
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/issues/57387#note_142682602
* [x] @yguo https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25293#note_142316006
* [ ] https://gitlab.com/gitlab-org/gitlab-ee/issues/5348#note_142649706
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25230#note_142474150
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25278#note_142134862
* [x] https://gitlab.com/gitlab-org/gitlab-ce/issues/57663#note_141096543
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21197#note_139607418
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25107#note_140670702 (Nik: checking it on a "restore" box)
* [ ] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25182#note_143426559
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25432#note_143741434
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/issues/57284#note_143632158
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25241#note_143637255
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7361#note_143467417
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9615#note_143467165

Story points: 2.0
Add a GKE cluster to gitlab-org/design.gitlab.com

# Why/What

We want to deploy `design.gitlab.com` using Auto DevOps (https://gitlab.com/gitlab-org/design.gitlab.com/issues/96). In order to do that we need a Kubernetes cluster associated with this project.

Probably this GKE cluster should live in our production GCP project but it could just as well live in a separate GCP project if that's preferable.

During the setup and probably for regular maintenance and troubleshooting we will need to provide some level of access to the GKE cluster for members of the Configure team (at least @DylanGriffith and @tauriedavis) as dogfooding Auto DevOps is one of our long term objectives. Also we assume that the production team will not want to be fully responsible for keeping this system up and running as it's not production critical infrastructure.

# How

1. Decide which GCP project to use
1. Somebody with permission to create GKE clusters in this GCP project will need to add a cluster to the https://gitlab-org/design.gitlab.com project (`Operations > Kubernetes > Add Kubernetes cluster > Sign in with Google`)
1. Name the cluster `design-system`
1. Leave the default settings for everything else and click create

Story points: 5.0
tflint is not configured property for CI/CD

Duplicated issue opened on ops instance [here](https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/issues/1)

Currently on stage `tf_validate` we have:

```
/bin/sh -e -c 'for d in $1; do echo "Checking $d for $0" && /terraform validate $2 "$d" && tflint --error-with-issues $2; done' "$env" "$dirs_to_check" "$tf_opts"
```

`$2` is `"$tf_opts"`, which evaluates `-check-variables=false` (which is not a `tflint` option as far as I can tell). This translates then in practice to running `tflint --error-with-issues` from the repo's root directory. If you try that from your local copy, you'll see it passes. But if you try it from, say `environments/gstg` you'll see errors. We should adjust this, since it's not actually doing any checks now (it seems).

/cc @alejandro

Story points: 3.0
Slack access for a Core Team member

While we work through the Slack access [issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6110) in general for all Core Team members, could we provided new Core Team member Ben Bodenmiller access to the following Slack channels?  Ben has already signed the NDA.

*  core
*  contribute2019
*  development
*  gdk
*  mr-coaching
*  release-post
*  security

Please let me know what you need from me.  I can send Ben's email address via DM on Slack.

cc/ @Mowry

Story points: 1.0
Execute maintenance on GitLab.com

Execute repacking maintenance on GitLab.com for indexes and selected tables (see below).

For selected tables, we also want to execute table maintenance. Those tables are the outliers based on the analysis in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/18277.

Those tables are:
* `services`
* `snippets`

The maintenance should be rolled out iteratively (with increased risk/impact):

1. Repack normal index
1. Repack primary key
1. Repack full table

#### User creation

* Create specific user for repacking
* Set `idle_in_transaction_timeout`

Story points: 8.0
Monitor bloat estimate

We need to measure bloat in the database to remove the need to manually measure it. Bloat can only be measured accurately on a one-off database and is a offline and rather expensive operation. Hence, we aim to estimate bloat with standard methods and push this information to prometheus.

Ideally, the bloat estimate should be available in prometheus for
* individual indexes,
* individual tables (heap + toast),
* overall index bloat (can be derived) and
* overall table bloat (can be derived).

It's ok if the bloat estimate only gets updated a few times a day, if a high frequency of measurements is not feasible.

Story points: 0.0
Set up Geo database replication for DR

Working database replication is a prerequisite for enabling geo. The DR site does not use omnibus bundled Postgres, so we need to get the replication set up manually and using Chef.

Currently, the DR nodes are running with unconfigured databases:
```shell
$ ssh devin-db@console-01-sv-dr.c.gitlab-dr.internal
Starting console, please wait ...
psql: FATAL:  no pg_hba.conf entry for host "10.251.16.2", user "gitlab", database "gitlabhq_production", SSL on
FATAL:  no pg_hba.conf entry for host "10.251.16.2", user "gitlab", database "gitlabhq_production", SSL off
Secondary check for db host patroni-03-db-dr.c.gitlab-dr.internal failed!
Connection to console-01-sv-dr.c.gitlab-dr.internal closed.
```

The DR database nodes are these:
```shell
$ knife node list | grep patroni | grep gitlab-dr
patroni-01-db-dr.c.gitlab-dr.internal
patroni-02-db-dr.c.gitlab-dr.internal
patroni-03-db-dr.c.gitlab-dr.internal
```

The next step is to load the data which was dumped from production into the DR cluster.  We want to do this at first without actually making a connection to production. This will reduce the risk.  Ideally, we should be able to load the database up to a certain time, and then switch on replication to catch up the difference, rather than turning on replication and having it pull the entire database from production directly.

Currently the DR database is minimally configured by terraform and chef.  If something goes wrong, it is trivial to delete the 3 nodes and re-create them from scratch.

Story points: 3.0
Automate Ansible runs for Gitter environment

Currently, we have to manually initiate Ansible runs in the Gitter environment to effect changes, such as [deploying SSH keys for newhires](gitlab-com/gl-infra/infrastructure#6171). We should at a minimum setup a service account to perform automated runs of `ansible-playbook` within a pipeline on gitlab-com/gl-infra/gitter-infrastructure>.

Other options possibly worth (future) discussion could include scheduled ansible runs, scheduled auto-scaling to reap older nodes and (re-)provision new ones, or even migration to an architecture based on immutable infrastructure (docker/k8s)

/cc @MadLittleMods @andrewn @sdval

Story points: 2.0
get familiar with omnibus setup for postgresql and repmgr

Please get more familiar with the setup for postgresql and rep mgr, looking forward at the integration of Patroni on the product.

Omnibus PG : https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/config/software/postgresql.rb

Marin: Docs for HA https://docs.gitlab.com/ee/administration/high_availability/database.html#doc-nav

Story points: 3.0
Transfer tanuki.cloud to GitLab

`tanuki.cloud` is a domain I registered in my early days @ GitLab in 2017 as we had a need for a domain to point to the Solution Architect AWS account.  I realized this is still on my personal account, and I'd rather transfer ownership to the company as the domain is GitLab's.

Let me know what other information you need from me to get started.

Story points: 2.0
Increase in support requests due to `repository_read_only`

There's been a recently influx of support requests for projects that are showing up as read only for users of GitLab.

Use this issue to figure out the overall impact to GitLab.com and investigate a way to figure out what might have flipped these projects into a read only state.

## Related
* https://gitlab.com/gitlab-com/support/dotcom/dotcom-escalations/issues/47
* https://gitlab.com/gitlab-org/gitlab-ce/issues/57263

## Other notes
* Initial cause was pointed at work being done here: https://gitlab.com/gitlab-com/gl-infra/production/issues/664
  * However, this work did not encompass any project from the above mentioned issues
  * I have logs for all projects I've touched during this work
  * The above projects are not in the logs, and are located on server to which the above maintenance is NOT being performed on

Story points: 1.0
Gemnasium service backup - restore test leaves db behind

Following https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4218, we found that the clean up of restored DBs doesn't work.
(see also https://gitlab.slack.com/archives/C8S0HHM44/p1549315690766800).

The initial clean (`destroy` function) doesn't look for remainings `production-restore-*` DBs, just for a DB with the current date (https://gitlab.com/gitlab-org/security-products/gemnasium/gcp-config/blob/cd7a218b7c951632152d40a5952dcdd9232d820f/bin/restore-test.sh#L127).
This script should be updated to list all `production-restore-*` DBs, and iterate on them to destroy each one.

/cc @skarbek

Story points: 2.0
Missing production_json logs from Kibana

It seems we have a problem ingesting particular logs from production_json.log from api nodes.

Here we have 4:

![graphql-logs](/uploads/7f1f61fe9942ce1d2c9ba7a8f69e1650/graphql-logs.png)

but locally we can see there are more:

```
$ knife ssh roles:gprd-base-fe-api "sudo zcat /var/log/gitlab/gitlab-rails/production_json.log.2.gz | jq -c 'select(.status == 500) | {time: .time, c: .controller}'"
api-01-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:46:52.933Z","c":"GraphqlController"}
api-01-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:47:08.689Z","c":"GraphqlController"}
api-01-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:48:57.082Z","c":"GraphqlController"}
api-01-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:49:03.639Z","c":"GraphqlController"}
api-18-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:46:59.982Z","c":"GraphqlController"}
api-18-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:47:01.723Z","c":"GraphqlController"}
api-18-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:47:10.095Z","c":"GraphqlController"}
api-18-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:47:50.679Z","c":"GraphqlController"}
api-13-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:48:01.023Z","c":"GraphqlController"}
api-13-sv-gprd.c.gitlab-production.internal     {"time":"2019-02-02T13:48:30.589Z","c":"GraphqlController"}
[snip]
$ knife ssh roles:gprd-base-fe-api "sudo zcat /var/log/gitlab/gitlab-rails/production_json.log.2.gz | jq -c 'select(.status == 500) | {time: .time, c: .controller}'" | wc -l
193
```
```

Story points: 2.0
Repositories with inconsistent move sizes

In this issue, we are moving data from nfs-file21 over to nfs-file25.  Two projects have data size inconsistencies between the two servers.  For both repos (just the data not the wiki's), the size on disk is larger on the location we are migrating too.  I'm curious if we can figure out potential corrupted data, and how to potentially coalesce anything if necessary.  At this moment in time, the database still points the storage of these repos to the old server `nfs-file21`.  These repos are currently marked as read only.  The goal for this issue, would be to remove what was moved to nfs-file25 (if safe), and mark the repositories as writable again.

## Projects in question:
* [ ] 9413811
* [ ] 9495429

## Reference Material
* https://gitlab.com/gitlab-com/gl-infra/production/issues/664#note_137118144

/cc @dawsmith

Story points: 2.0
Most requests in staging time out

Staging is throwing 502 errors on most web requests.

Unicorn processes are being killed:

![Screen_Shot_2019-01-31_at_2.12.39_PM](/uploads/d1ba8db3ae88c656633fd2ce618e32ea/Screen_Shot_2019-01-31_at_2.12.39_PM.png)

Workhors is throwing 502 errors:

![Screen_Shot_2019-01-31_at_2.13.49_PM](/uploads/2532778d4f74013ea1d3e64aa75e07ef/Screen_Shot_2019-01-31_at_2.13.49_PM.png)

The problem seems to be database related:

![Screen_Shot_2019-01-31_at_2.16.00_PM](/uploads/64b698d6b25a8918d5d9588aed56e3c6/Screen_Shot_2019-01-31_at_2.16.00_PM.png)

Queries are timing out:

![Screen_Shot_2019-01-31_at_2.21.06_PM](/uploads/46401efc3a6ff942f5bb29d19428e9d5/Screen_Shot_2019-01-31_at_2.21.06_PM.png)

And transactions are very high on patroni-06:

![Screen_Shot_2019-01-31_at_2.23.44_PM](/uploads/b4cfaf17fc9d83b3524c9045531332eb/Screen_Shot_2019-01-31_at_2.23.44_PM.png)

Story points: 1.0
Investigate and Design Execution Plan for Migrating Chef server to GCP

As outcome of #6066 we want to move the Chef server from Digital Ocean to GCP.
We should test restoring Chef from backups and switch from Ubuntu 14.04 to 18.04 and then consider if we also want to upgrade Chef to a newer version.

Chef should run in a different region of the `gitlab-ops` project to be able to bootstrap GitLab.com in case other regions are down.

1. [ ] Document, plan, and test backup & restore process to be used in production change for migration [#8028]
1. [ ] Identify/plan updates required to `chef-repo` pipelines and related workflow processes (cookbooks, roles, environments, data bags, vaults, ...others?) to support the new chef server [#8036]
1. [x] Provision infrastructure for new chef server(s) in `gitlab-ops` project [#8037]
1. [ ] Document and perform test-run(s) of the migration (no staging infrastructure available)

Story points: 3.0
Functional onboarding buddy

Currently, a new hire is assigned an *onboarding buddy* but usually they're not in the same functional role as the new hire (on purpose). However, having somebody in the same functional role dedicated to your onboarding can help to speed up getting started.

We have a lot of onboarding material and having a functional onboarding buddy is not meant to replace that. We want to stay async when onboarding. However, for certain topics, it does help to have somebody dedicated to point in the right direction or provide a short talk about a certain topic. At minimum, the functional onboarding buddy would aim to convey which topics are important to understand and provide starting points.

This is an outcome from the DBRE sync meeting today, suggestion by @yguo and @cshobe .

Story points: 1.0
Database Reviews

* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9182
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8949
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24198 and https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9219#2cda0c6171fa7d04989507a1dd112e34c40df46d
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24377#note_135696943 ~"Community Contribution" 
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24458#note_135715085 ~"Community Contribution" 
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24599#note_134906250
* [x] @NikolayS  https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9283#note_136104503
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24743
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8988#note_136809963 (inherited from the previous weeks)
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9267
* [x] https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24677#note_137415396
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24803#note_137093489
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9364#note_137035887
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580#note_136674989
* [ ] @NikolayS https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21197#note_136377797
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9283#note_136104503
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9332#note_136032973
* [x] @NikolayS https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9334
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24881
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24677#note_137946482
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9398/diffs
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23596
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9240#6f917fe927dcf9c1ca3635688e5ef4dca229d9eb
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9420#note_138504910 together with https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24822
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24923#note_138262555
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9267
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24822
* [x] @abrandl https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22743#4df3c9d1a5b5d7ae199b2914f6b019ceefbd64ae

Story points: 2.0
http://stats.gitlab.com shows no data

http://stats.gitlab.com shows no data. If this is correct, we should decom this site.![Screen_Shot_2019-01-27_at_1.26.20_AM](/uploads/89fe454b891b5807bcbb42cbc5072075/Screen_Shot_2019-01-27_at_1.26.20_AM.png)

Story points: 1.0
Increase web worker pool

Based on peaks in our monitoring data, we're hitting the limit of how many web requests we can handle at any given time.

![web-worker-saturation-2018-01-25](/uploads/30f738a8e46d89fb6cb238c568cac661/web-worker-saturation-2018-01-25.png)

For reference we have peak-time CPU utilization of 65% and memory utilization of 45%

![web-fleet-cpu-2018-01-25](/uploads/abc354f59459fc76bcbbf057cc49ae3b/web-fleet-cpu-2018-01-25.png)

![web-fleet-memory-2018-01-25](/uploads/350f9e2afdacf3a7ab486a7f9aa3d341/web-fleet-memory-2018-01-25.png)

We currently deploy 30 workers to 16 cpu nodes. I'm thinking a 10% increase to 36/node (20% more workers) would stay within acceptable CPU and memory utilization.

Story points: 2.0