Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?

Discussion:
The example combines a web frontend, a redis master for storage and a replicated set of redis slaves. ### Step Zero: Prerequisites <del> This example assumes that you have forked the repository and [turned up a Kubernetes cluster](https://github.com/GoogleCloudPlatform/kubernetes-new#setup). </del> <ins> This example assumes that you have forked the repository and [turned up a Kubernetes cluster](https://github.com/GoogleCloudPlatform/kubernetes-new#setup): $ cd kubernetes $ hack/dev-build-and-up.sh </ins> ### Step One: Turn up the redis master. <del> Create a file named `redis-master.json`, this file is describes a single pod, which runs a redis key-value server in a container. </del> <ins> Create a file named `redis-master.json` describing a single pod, which runs a redis key-value server in a container. </ins> ```javascript {

} ``` <del> Once you have that pod file, you can create the redis pod in your Kubernetes cluster using the `cloudcfg` cli: </del> <ins> Once you have that pod file, you can create the redis pod in your Kubernetes cluster using the `cloudcfg` CLI: </ins> ```shell <del> cluster/cloudcfg.sh -c redis-master.json create /pods </del> <ins> $ cluster/cloudcfg.sh -c examples/guestbook/redis-master.json create /pods </ins> ``` Once that's up you can list the pods in the cluster, to verify that the master is running:

cluster/cloudcfg.sh list /pods ``` <del> You should see a single redis master pod.  It will also display the machine that the pod is running on.  If you ssh to that machine, you can run </del> <ins> You'll see a single redis master pod. It will also display the machine that the pod is running on. ```javascript Using master: kubernetes-master (external IP: 1.2.3.4) { "kind": "cluster#podList", "items": [ { "id": "redis-master-2", "labels": { "name": "redis-master" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "name": "master", "image": "dockerfile/redis", "ports": [ { "hostPort": 6379, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } } ] } ``` If you ssh to that machine, you can run `docker ps` to see the actual pod: </ins> ```shell <del> sudo docker ps </del> <ins> $ gcloud compute ssh kubernetes-minion-3 --zone us-central1-b $ sudo docker ps me@kubernetes-minion-3:~$ sudo docker ps CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES 417ab993cdf8  dockerfile/redis:latest  redis-server /etc/re  8 minutes ago Up 8 minutes  0.0.0.0:6379->6379/tcp  master--redis_-_master_-_2--6b944b49 </ins> ``` <del> And see the actual pod.  (Note that initial `docker pull` may take a few minutes, depending on network conditions.) </del> <ins> (Note that initial `docker pull` may take a few minutes, depending on network conditions.) </ins> ### Step Two: Turn up the master service. <del> A Kubernetes 'service' is a named load balancer that proxies traffic to one or more containers.  The services in a Kubernetes cluster are discoverable inside other containers via environment variables.  Services find the containers to load balance based on pod labels.  The pod that you created in Step One has the label `name=redis-master`, so the corresponding service is defined by that label.  Create a file named `redis-master-service.json` that contains: </del> <ins> A Kubernetes 'service' is a named load balancer that proxies traffic to one or more containers. The services in a Kubernetes cluster are discoverable inside other containers via environment variables. Services find the containers to load balance based on pod labels. </ins> <del> ```javascript </del> <ins> The pod that you created in Step One has the label `name=redis-master`, so the corresponding service is defined by that label.  Create a file named `redis-master-service.json` that contains: ```js </ins> { "id": "redismaster", "port": 10000,

Once you have that service description, you can create the service with the `cloudcfg` cli: <del> ```shell cluster/cloudcfg.sh -c redis-master-service.json create /services </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-master-service.json create /services Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redismaster", "port": 10000, "labels": { "name": "redis-master" } } </ins> ``` Once created, the service proxy on each minion is configured to set up a proxy on the specified port (in this case port 10000). ### Step Three: Turn up the replicated slave pods. <del> Although the redis master is a single pod, the redis read slaves are a 'replicated' pod, in Kubernetes, a replication controller is responsible for managing multiple instances of a replicated pod.  Create a file named `redis-slave-controller.json` that contains: </del> <ins> Although the redis master is a single pod, the redis read slaves are a 'replicated' pod. In Kubernetes, a replication controller is responsible for managing multiple instances of a replicated pod. </ins> <del> ```javascript </del> <ins> Create a file named `redis-slave-controller.json` that contains: ```js </ins> { "id": "redisSlaveController", "desiredState": {

Then you can create the service by running: <del> ```shell cluster/cloudcfg.sh -c redis-slave-controller.json create /replicationControllers </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-slave-controller.json create /replicationControllers Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redisSlaveController", "desiredState": { "replicas": 2, "replicasInSet": { "name": "redisslave" }, "podTemplate": { "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "labels": { "name": "redisslave" } } }, "labels": { "name": "redisslave" } } </ins> ``` The redis slave configures itself by looking for the Kubernetes service environment variables in the container environment.  In particular, the redis slave is started with the following command:

Once that's up you can list the pods in the cluster, to verify that the master and slaves are running: <del> ```shell cluster/cloudcfg.sh list /pods </del> <ins> ```js $ cluster/cloudcfg.sh list /pods Using master: kubernetes-master (external IP: 23.236.49.160) { "kind": "cluster#podList", "items": [ { "id": "redis-master-2", "labels": { "name": "redis-master" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "name": "master", "image": "dockerfile/redis", "ports": [ { "hostPort": 6379, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "4d65822107fcfd52", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "78629a0f5f3f164f", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-2.c.{project-id}.internal" } } ] } </ins> ``` <del> You should see a single redis master pod, and two redis slave pods. </del> <ins> You will see a single redis master pod and two redis slave pods. </ins> ### Step Four: Create the redis slave service. Just like the master, we want to have a service to proxy connections to the read slaves.  In this case, in addition to discovery, the slave service provides transparent load balancing to clients.  As before, create a service specification: <del> ```javascript </del> <ins> ```js </ins> { "id": "redisslave", "port": 10001,

This time the label query for the service is `name=redis-slave`. <del> Now that you have created the service specification, create it in your cluster with the `cloudcfg` cli: </del> <ins> Now that you have created the service specification, create it in your cluster with the `cloudcfg` CLI: ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-slave-service.json create /services Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redisslave", "port": 10001, "labels": { "name": "redisslave" } } </ins> <del> ```shell cluster/cloudcfg.sh -c redis-slave-service.json create /services </del> ``` ### Step Five: Create the frontend pod. <del> This is a simple PHP server that is configured to talk to both the slave and master services depdending on if the request is a read or a write.  It exposes a simple AJAX interface, and serves an angular based U/X.  Like the redis read slaves it is a replicated service instantiated by a replication controller.  Create a file named `frontend-controller.json`: </del> <ins> This is a simple PHP server that is configured to talk to either the slave or master services depending on whether the request is a read or a write. It exposes a simple AJAX interface, and serves an angular-based UX. Like the redis read slaves it is a replicated service instantiated by a replication controller. </ins> <del> ```javascript { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": {"name": "frontend"}, "podTemplate": { "desiredState": { "manifest": { "containers": [{ "image": "brendanburns/php-redis", "ports": [{"containerPort": 80, "hostPort": 8080}] }] } }, "labels": {"name": "frontend"} }}, "labels": {"name": "frontend"} } </del> <ins> Create a file named `frontend-controller.json`: ```js { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": {"name": "frontend"}, "podTemplate": { "desiredState": { "manifest": { "containers": [{ "image": "brendanburns/php-redis", "ports": [{"containerPort": 80, "hostPort": 8080}] }] } }, "labels": {"name": "frontend"} }}, "labels": {"name": "frontend"} } </ins> ``` With this file, you can turn up your frontend with: <del> ```shell cluster/cloudcfg.sh -c frontend-controller.json create /replicationControllers </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/frontend-controller.json create /replicationControllers Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "frontendController", "desiredState": { "replicas": 3, "replicasInSet": { "name": "frontend" }, "podTemplate": { "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "labels": { "name": "frontend" } } }, "labels": { "name": "frontend" } } </ins> ``` Once that's up you can list the pods in the cluster, to verify that the master, slaves and frontends are running: <del> ```shell cluster/cloudcfg.sh list /pods </del> <ins> ```js $ cluster/cloudcfg.sh list /pods Using master: kubernetes-master (external IP: 1.2.3.4) { "kind": "cluster#podList", "items": [ { "id": "redis-master-2", "labels": { "name": "redis-master" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "name": "master", "image": "dockerfile/redis", "ports": [ { "hostPort": 6379, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "4d65822107fcfd52", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "55104dc76695721d", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } }, { "id": "78629a0f5f3f164f", "labels": { "name": "redisslave", "replicationController": "redisSlaveController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-2.c.{project-id}.internal" } }, { "id": "380704bb7b4d7c03", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-2.c.{project-id}.internal" } }, { "id": "365a858149c6e2d1", "labels": { "name": "frontend", "replicationController": "frontendController" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/php-redis", "ports": [ { "hostPort": 8080, "containerPort": 80 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-4.c.{project-id}.internal" } } ] } </ins> ``` <del> You should see a single redis master pod, two redis slave and three frontend pods. </del> <ins> You will see a single redis master pod, two redis slaves, and three frontend pods. </ins> The code for the PHP service looks like this: ```php <?

} ?> ``` <del> To play with the service itself, find the name of a frontend, grab the external IP of that host from the [Google Cloud Console][cloud-console], and visit `http://<host-ip>:8080`. Note, you may need to open the firewall for port 8080 using the [console][cloud-console] or the `gcloud` tool. </del> <ins> To play with the service itself, find the name of a frontend, grab the external IP of that host from the [Google Cloud Console][cloud-console], and visit `http://<host-ip>:8080`. You may need to open the firewall for port 8080 using the [console][cloud-console] or the `gcloud` tool. </ins> [cloud-console]: https://console.developer.google.com