Issue 1:
This is a roundup issue for Kubernetes 1.4 and a tracker to orient folks on the ongoing work. It is focused on all components that touch the API, but not necessarily the total system. It also serves to identify areas where folks can get involved.

Track issues using labels performance and CSI-API-Machinery

Ongoing efforts

Major focus: We want to get to our target cluster size metrics (5k nodes and 300k pods) in 1.4 or 1.5, and we are within an order of magnitude of that with 1.3 in many dimensions.

Areas of attack: We are mostly CPU bound on the controllers and API servers today, and most of that is overhead due to garbage collection. We also have had a fair number of contention areas that were algorithmic or lock related in the core components. We've got a good feedback loop for identifying critical bottlenecks, but we could probably stand to get more profiles and metrics gathered from the high scale systems.

Reduce the CPU and garbage created by clients and the API server when handling API requests

Introduce protobuf

[complete #8132] Used for most API objects when talking between client and server

[inprogress for 1.5] Use protobuf for storage format

Handle mixed content type storage (objects that embed other objects like Templates or third party resources)

[inprogress #27258] Handle optional slices and maps correctly in protobuf

inprogress Investigate use of string arena unmarshalling in protobuf to avoid []byte -> string allocations in decode

Optimize the conversion, defaulting, and decoding code paths

complete Implement more efficient Quantity, using int64 and better serialization

[complete #24966] Remove the use of reflection in conversion and defaulting

Auto generate fast path conversion functions (AddGenericConversionFunc) for each API type

[complete #24727] Make conversions of identical types (builtins, maps, slices, pointers) an assignment, not a copy

[inprogress #27659] Further optimization of identical type conversion using unsafe.Pointer

[inprogress #33352] Split defaulting from conversion into its own function, possibly by generating an entire "Default_Pod" function from component parts, which allows us to avoid having to walk deep objects

Use the versioned client where possible to avoid conversion altogether. #29934

inprogress Versioned client in controllers depends on the shared cache work (all controllers sharing a cache have to use the same object type). It's also blocked by the cutting a client library work.

Ensure that the Update and Create code paths are efficient (guaranteed update loop generates a lot of objects)

SelfLink and URL.String() are fairly expensive in the encode/decode path

For any JSON gaps (non-proto clients) we could look at easyjson, which appears to do more than ugorgji to reduce memory allocation during decode.

Reduce the amount of duplicate objects being used by controller and admission caches and the amount of total memory used by caches

[complete] Separate re-listing and re-syncing (former talks to api server, latter traverses cache) to reduce the amount of duplicate calls to the server and wasted conversion

complete Add a SharedInformer and use it for Pod caches (all 8 of them)

[inprogress] Use shared Namespace caches in admission controllers

Investigate and identify other opportunities for shared caches

[inprogress] Investigate size of objects kept in shared and unshared caches for duplication and waste

Pods tend to have larger objects https://github.com/coreos/kscale/pull/37/files#diff-a423e914cbd27c95d2c0b950e2e4ee5bR14 than necessary - maps and quantity too large

Investigate using slices instead of maps for our many small maps, or having an ArrayMap equivalent to Java #31440

Investigate flyweight cache patterns for certain controllers

Identify unnecessary blocking / stalls in clients, watches, caches, and controllers

[complete] Several fixes to 1.3 scheduler and caches

[complete] Avoid flushing aggressively in watches to reduces syscalls

inprogress Shared caches currently have coarse locks, investigate striping and reducing the duration of the locks

Investigate whether we have any slow cache reader problems with our shared caches

Audit the watch cache for locking issues

Better tooling and documentation to allow people to perform this sort of debugging

???

Profile and optimize specific components and workloads

Collect profile traces from kubemark and high scale clusters automatically

Scheduler (lots of eyes on this, but also lots of algorithmic improvements possible)

Kubelet

[inprogress] Replication controller

Node controller (known issues with Timer, other garbage creation)

Quota

[inprogress] Significant improvements including contention striping and batch assignment

Optimize and audit use of resource.Quantity in these codepaths

Authentication / Authorization / RBAC (has been a significant source of perf bottlenecks in OpenShift in the more complex scenarios)

???

Reduce traffic between components

Investigation colocation of certain processes if necessary

OpenShift offers all-in-one processes that bundle etcd, controller, apiserver, as does a few others, and there is some value in compact deployments having an efficient master (downsides are increased memory use and contention)

inprogress Investigate loopback connections when in the same process (at the net.Chan level)

[mentioned by justinsb] Directly embed etcd and make calls to the Go API

Reduce network bandwidth by sending less objects (not yet a blocker in any particular area)

Issue 2:
There is a lot of copied bash between the various providers. We should create a shared library that multiple providers can use that'll simplify the amount of code for each provider.