Kubernetes 1.18 broke “kubectl run”, here’s what to do about it
By: Alex Ellis
Tags: Docker, Kubernetes, Developers, Cloud Computing, Open Source

If you’ve been using a recent release of Kubernetes like 1.16, then you may have been used to seeing an odd warning message from kubectl run . Up until recently, the command could be used to generate Deployment objects or YAML from the command line.

With Kubernetes 1.18 the user experience for generating and creating Deployments has been broken in a surprising way. Here’s why, and what you can do about it.

Before we get into the change itself, there are three primary ways to create a Deployment object in Kubernetes:

How to create a Deployment in Kubernetes

Using the REST API and a client such as client-go (this is what OpenFaaS does)

By hand-crafting a YAML file, or more realistically, by copying and pasting from StackOverflow or a known example

By kubectl run or kubectl run -o yaml --dry-run

It turns out that I also listed these options in order of complexity (high to low).

Comparing each option

For Option 1 — using the REST API is probably one of the most stable ways to create Deployments, but is the most complex option you can find. Up until recently the API had been relatively stable and happy in “extensions/v1beta” until it moved to “apps/v1”, this was a breaking change and meant lots of OSS projects having to migrate their code and YAML files.

I am not sure what’s harder — crafting Kubernetes objects in Go, or in YAML. With Go you get some hints in the IDE, but have to search all over for which package each type exists in. Core? Meta? Apps? Extensions? Maybe that’s what we’re missing with editing manifest files in VSCode — some good old intellisense?

A snippet of code from the OpenFaaS controller for Kubernetes (faas-netes)

Option 2 is hand-crafting YAML. If you’ve ever used Twitter or watched the #kubernetes hashtag, you’ll see all kinds of folks from newcomers to respected Kubernetes authors and founders bashing on how hard it is to write and grok YAML. Personally I like YAML and find it easy to use in most applications, but I do see where they are coming from. I think what they are really complaining about is the complexity and nested structure of the Kubernetes API schema.

apiVersion: apps/v1

kind: Deployment

metadata:

name: nginx-1

labels:

app: nginx

spec:

replicas: 1

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:1.14.2

ports:

- containerPort: 80

This example was actually copied from the Kubernetes docs and adjusted to set the replicas to 1/1 instead of 3. I personally would hate to be put on the spot and asked to write this from scratch.

Writing Kubernetes YAML from scratch feels like having to reverse a double-linked list with a pencil on a piece of paper and then being told not to make any mistakes or typos.

Option 3 was until recently a fine alternative to 1 and 2, especially when learning, when wanting succinct instructions to demonstrate another application or controller, or when wanting to generate a sample YAML file for further customisation or for use in a CI/CD pipeline.

To create a Deployment directly into the cluster, with a restart policy, a name, an image and a port, you could do this:

$ kubectl run nginx-1 --image=nginx --port=80 --restart=Always

Even better, if you wanted to mix option 2 and 3, you could output YAML without applying it and then customise it. Let’s see how it compares to the documentation?

$ kubectl run nginx-1 --image=nginx:1.14.2 --port=80 \

--restart=Always -o yaml --dry-run apiVersion: apps/v1

kind: Deployment

metadata:

creationTimestamp: null

labels:

run: nginx-1

name: nginx-1

spec:

replicas: 1

selector:

matchLabels:

run: nginx-1

strategy: {}

template:

metadata:

creationTimestamp: null

labels:

run: nginx-1

spec:

containers:

- image: nginx:1.14.2

name: nginx-1

ports:

- containerPort: 80

resources: {}

Other flags you could set included --replicas and --serviceaccount to name a couple.

It’s not too dissimilar from the documentation, but the reason I’m writing this post is to let you know that this feature has been permanently removed in kubectl 1.18 and later.

Warning signs

I was always a little confused by this error, but the deprecation notice didn’t seem to have a time-line or any other helpful context.

Like you, I just thought “OK so I’ll learn a different CLI command at some point in the near future”

$ kubectl run nginx-1 --image=nginx --port=80 --restart=Always kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/nginx-1 created

If you download kubectl 1.17 or earlier (I am using 1.15 above) you’ll still be able to create a deployment or generate from from the CLI. As soon as you switch over to Kubernetes 1.18, you’ll get the following instead:

kubectl run nginx-1 --image=nginx --port=80 --restart=Always -o yaml --dry-run

W0512 14:27:13.111424 30104 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client. apiVersion: v1

kind: Pod

metadata:

creationTimestamp: null

labels:

run: nginx-1

name: nginx-1

spec:

containers:

- image: nginx

name: nginx-1

ports:

- containerPort: 80

resources: {}

dnsPolicy: ClusterFirst

restartPolicy: Always

That’s a Pod, and it’s not the same as a Deployment. Fortunately you can still specify a service account and a port, but working directly with Pods is not encouraged as first-class citizens when building a production application. Pods are ephemeral, they can move around, get killed, go to OOM and don’t have a stable name (by default) or IP address. Deployments and Services fix this problem.

Pods are more useful for one-shot tasks like running curl to check your API or for adding some messages to a Kafka topic.

We faced this first-hand on the inlets-operator project, when a user told us that the documentation no longer worked.

You can expose a Pod, if you wish with “kubectl expose pod”.

For our documentation and helm-chart helper message, we wanted a simple instruction that would help users understand how to use the project. You create a Deployment, then expose it as a LoadBalancer Service, we provision a VM on your favourite cloud and then tunnel it back, so you in effect have a fully integrated public IP.

Our documentation pre-k8s 1.17 was:

kubectl run nginx-1 --image=nginx --port=80 --restart=Always

kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer

It’s now:

export DEPLOYMENT=nginx-1 (cat<<EOF

apiVersion: apps/v1

kind: Deployment

metadata:

name: $DEPLOYMENT

labels:

app: nginx

spec:

replicas: 1

selector:

matchLabels:

app: nginx

template:

metadata:

labels:

app: nginx

spec:

containers:

- name: nginx

image: nginx:1.14.2

ports:

- containerPort: 80

EOF

) | kubectl apply -f - kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer

Note to the Kubernetes maintainers, please don’t take kubectl expose away from us next :-)

I am not looking forward to maintaining that amount of text in every tutorial I write, or whenever I need to craft a new Deployment.

I was so unhappy with the resulting documentation, that I’ve moved that over to an embedded file, which people will now need to first download to check, before running into their cluster:

$ kubectl apply -f https://raw.githubusercontent.com/inlets/inlets-operator/master/contrib/nginx-sample-deployment.yaml

Folks, never kubectl apply -f files from the Internet, without checking them over first.

So what’s the alternative?

It turns out that there is no equivalent alternative.

Some will be quick to tell you that the new command is kubectl create deployment.

However you’ll notice that all the important flags are missing like --port , --serviceaccount , --replicas and --restart-policy .

kubectl create deployment --help

Create a deployment with the specified name. Aliases:

deployment, deploy Examples:

# Create a new deployment named my-dep that runs the busybox image.

kubectl create deployment my-dep --image=busybox

--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in

the template. Only applies to golang and jsonpath output formats.

--dry-run='none': Must be "none", "server", or "client". If client strategy, only print the object that would be

sent, without sending it. If server strategy, submit server-side request without persisting the resource.

--image=[]: Image name to run.

-o, --output='': Output format. One of:

json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.

--save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the

annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.

--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The

template format is golang templates [

--validate=true: If true, use a schema to validate the input before sending it Options:--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing inthe template. Only applies to golang and jsonpath output formats.--dry-run='none': Must be "none", "server", or "client". If client strategy, only print the object that would besent, without sending it. If server strategy, submit server-side request without persisting the resource.--image=[]: Image name to run.-o, --output='': Output format. One of:json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.--save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, theannotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. Thetemplate format is golang templates [ http://golang.org/pkg/text/template/#pkg-overview ].--validate=true: If true, use a schema to validate the input before sending it Usage:

kubectl create deployment NAME --image=image [--dry-run=server|client|none] [options] Use "kubectl options" for a list of global command-line options (applies to all commands).

So our original example will no longer work with kubectl create , and cannot be ported to anything but a block of YAML. If we need a dynamic port, then the alternative is going to be some bash templating which is not cross-platform.

The best we can get to with the new command is this, but then you’re going to have to trawl StackOverflow again, or the Kubernetes docs every time you want a simple Deployment. Why? Because nested YAML schemas are hard to memorise.

$ kubectl create deployment nginx-1 --image=nginx -o yaml --dry-run W0512 14:38:18.296270 30135 helpers.go:535] --dry-run is deprecated and can be replaced with --dry-run=client. apiVersion: apps/v1

kind: Deployment

metadata:

creationTimestamp: null

labels:

app: nginx-1

name: nginx-1

spec:

replicas: 1

selector:

matchLabels:

app: nginx-1

strategy: {}

template:

metadata:

creationTimestamp: null

labels:

app: nginx-1

spec:

containers:

- image: nginx

name: nginx

resources: {}

Note the missing port, restart-policy and more.

What’s the lesson of the day?

I am not sure, but sometimes good intentions and dealing with nagging technical debt can result in a worse user-experience despite every hope of improving the world.

Kubernetes PR 68132 seemed to start it all, by wanting kubectl run to work more like the docker run command. Certainly not a bad thing.

My hope was that the maintainers would consider adding the various missing flags to the new kubectl create deployment command, however someone else got here first and was told “if you want it, you send a pull request”.

Did you get affected by this change? Comment on the Twitter thread, or the PR above which merged the change.

Making things better is difficult, especially when it breaks long established user-experiences. You do you, Kubernetes authors 💙.

For a complete list of changes in 1.18, see the release notes: https://kubernetes.io/docs/setup/release/notes/#changelog-since-v1170

You may also like: