Code change:
We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.
+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc

Related discussion:
text: <ins> <!-- BEGIN MUNGE: UNVERSIONED_WARNING --> <!-- BEGIN STRIP_FOR_RELEASE --> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> <img src="http://kubernetes.io/img/warning.png" alt="WARNING" width="25" height="25"> <h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2> If you are using a released version of Kubernetes, you should refer to the docs that go with that version. <strong> The latest release of this document can be found [here](http://releases.k8s.io/release-1.1/docs/proposals/choosing-scheduler.md). Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io). </strong> -- <!-- END STRIP_FOR_RELEASE --> <!-- END MUNGE: UNVERSIONED_WARNING --> Scheduler Performance Test ====== Motivation ------ We already have a performance testing system -- Kubemark. However, Kubemark requires setting up and bootstrapping a whole cluster, which takes a lot of time. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. We have the following goals: - Save time on testing - The test and benchmark can be run in a single box. We only set up components necessary to scheduling without booting up a cluster. - Profiling runtime metrics to find out bottleneck - Write scheduler integration test but focus on performance measurement. Take advantage of go profiling tools and collect fine-grained metrics, like cpu-profiling, memory-profiling and block-profiling. - Reproduce test result easily - We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. Currently the test suite has the following: - density test (by adding a new Go test) - schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes - print out scheduling rate every second - let you learn the rate changes vs number of scheduled pods - benchmark - make use of `go test -bench` and report nanosecond/op. - schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small: 10 - 100. How To Run ------ ``` cd kubernetes/test/component/scheduler/perf ./test-performance.sh ``` <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/test/component/scheduler/perf/README.md?pixel)]() <!-- END MUNGE: GENERATED_ANALYTICS --> </ins>