Configuring Default Resource Requests, Limits, and Quotas on your new OpenShift 4 Cluster
By: Benjamin Porter
Tags: Infrastructure As Code, Openshift 4, System Administration, Infrastructure, Openshift

There are numerous tasks that need to be done to ready a newly installed OpenShift cluster for use by developers. One such task is configuring default requests and limits, as well as project quotas.

Note: All examples here were developed/tested on OpenShift 4.5.5

Note: In bash shell code examples, the $ at the beginning is just used to indicate a command prompt. If you are copy/pasting commands with me, do not include the leading $ in your paste

Why?

Most developers will not set requests/limits on their project. This is not due to laziness or apathy. It’s more of an “out of sight, out of mind” problem. However, failure to do so makes the resources their pods are allowed to consume effectively unlimited, which as a system administrator is not what you want. This can allow some Pods and containers to starve the system of resources, which can negatively impact other teams and users of the system.

The Solution

You can configure the default project template to include requests and limits, such that if a developer forgets, they will be given a sensible default which can be tailored by the system administrator to be appropriate for the environment and cluster.

Because this template is only invoked for new projects, it’s important to note that it will not affect older projects that may be missing explicit requests and limits.

If the bootstrap project template has not yet been created, you can generate one with the oc tool:

$ oc adm create-bootstrap-project-template -o yaml > template.yaml

You can see that this is just a normal OpenShift Template. What makes it special is that this is the Template that will be used to create new projects, such as when a user runs oc new-project <project-name> .

The objects defined here will get created in each new project space.

Notice that there are some helpful variables already defined. We will make use of these with the new objects we are going to create.

The first object we’ll create is a LimitRange that will apply to each Pod and/or Container. Let’s define a LimitRange object that includes default settings. These will get applied to applied to any Pod which doesn’t explicitly declare its CPU and Memory requests:

In this Template we are saying that any Pods that don’t specify a request will be given half a CPU and 500Mi of memory. The limit will be set to 1 CPU and 1Gi of memory. Because the requests and limits are both set and they are not the same number, the Quality of Service here will be Burstable.

Next, we need to set a ResourceQuota for the new project. Unlike the LimitRange, the ResourceQuota applies to an entire namespace, and is the sum of all resources consumed. There are a number of different resources you can apply a quota to, but the most common ones are Requests/Limits CPU/Memory, storage, and number of Pods.

Let’s create a simple ResourceQuota that allows 10 Pods in the namespace, up to 6 CPUs, and 16Gi of memory.

Now that we have these two objects defined, we are ready to add them to the objects array in our project template. Here is our complete new project template that we will be setting as the default template for creating new projects:

Notice that we did not change anything in the previous template, other than adding our LimitRange and ResourceQuota to it.

Let’s go ahead and apply this to the cluster. Make sure to apply this to the correct namespace. It must to go into the openshift-config namespace.

The namespace can be specified either on the command line, or in the YAML file itself. I prefer the latter strategy personally, but since the generated Template does not include a namespace in the YAML, I followed suit and left it out and specified on the command line in this case:

$ oc apply -f template.yaml -n openshift-config

Lastly we need to configure the cluster to use our Template for bootstrapping new projects. Let’s do so using the oc patch command:

Your new project template is installed!

But as they say, the proof is in the pudding, so let’s try creating a new project and verify the new objects are created:

Great! If you get an error on project creation, try a few times and possibly try running the oc patch command again a few times too. For some reason I've seen it get stuck for a second or two and need another kick.

You can visually inspect the project config as well:

$ oc get project.config.openshift.io/cluster -o yaml

...

<YAML in here for you to inspect>

...

Make sure our template’s name “ project-request” is referenced at the bottom of the project config in the spec section:

Or just dump the whole project config to the terminal, and look for the spec section at the bottom:

$ oc get project.config.openshift.io cluster -o yaml

...

spec:

projectRequestTemplate:

name: project-request

Things to look out for

1. This can affect Quality of Service

The requests and limits on a Pod will affect Quality of Service (QoS), so you should familiarize yourself with that concept to avoid creating a policy that has undesired side effects. For example, you may want to avoid setting the request and limits to the same number. This causes your Pods to run with “Guaranteed” QoS, which may not be what you want for every Pod in the cluster.

2. Slightly complicated rules

There are quite a few “rules” of logic regarding requests and limits, mins and maxes. I would recommend you familiarize yourself with them by reading the documentation for limit ranges.

Some cluster administrators like to set the requests very low as a means of “requiring” developers to specify. If they do not specify, then their Pods will be starved of resources. However, this can cause a couple of problems, a classic example is:

A. Developers will be surprised and confused if limits are super low, and will waste development time troubleshooting

You will want to do a good job disseminating knowledge about the defaults. Even with a respectable dissemination effort, there will still be plenty of people that miss the news and are surprised when their Pod struggles to run. These developers may not know how to fix their problem (particularly as many developers know only the bare minimum about Kubernetes), and may waste considerable time spinning their wheels.

3. Strict quotas can prevent deploys

Quotas are useful tools to prevent overuse and abuse of the system, particularly in limited capacity settings. However, much of the time the point of discovery that you have fulfilled a quota limit is when you are trying to deploy your application. This can be a most inconvenient time to be wrangling quotas, particularly if the team managing the deployment does not have control of the quota! It is good to keep this in mind to avoid causing problems for development teams. I highly recommend continually monitoring quotas, so that when a namespace is close to filling up, the proper team can be alerted ahead of time rather than having their deploy fail.

Closing Thoughts

Default requests, limits, and quotas are not difficult to administer when one knows how. Hopefully this post has made it easy to understand what is going on.