Issue 1:
Our current API strategy is to add annotations for new features. These annotations are expected to overlay on top of the existing API fields, and alter system behavior in other components. Our primary reason for annotations today:

They are unstructured and can be preserved in etcd even after the feature is graduated from alpha.

They can contain arbitrarily complex values (like init containers) or simple fields (like hostname for pods)

Annotations come with a set of issues:

Alpha annotations are not discoverable and do not show up in our docs - users don't know how to use the features, and we get no automatic benefits from swagger or other tools

No way to communicate lifecycle via kubectl explain to end users

Developers who implement those annotations must merge the value in to the internal or external types at various points in the lifecycle - conversion, validation, in client libraries, and in nested utilities.

Client utilities that receive a partial struct (like api.PodSpec) must also receive a list of annotations that alter that pod spec (for things like init containers) and potentially merge those

When an alpha feature is graduated, we typically change the implementation of that feature significantly as well (moving from annotation to field) which causes extra churn in the implementation, and the potential for new bugs

Some alpha features may be missed by a developer unfamiliar with the annotation - without any of the normal tools to introspect code (code completion, reflective tests on structs) it is more likely that the impact of an annotation is missed when adding new code.

The security implications of an annotation are harder to convey than a field, and code that allows annotation mutation but not field annotation can be exposed to malicious authors accidentally (bind, status, and a few other sub resources that protect spec can be abused to mutate init containers, for example).

A malicious user can introduce an annotation for an upcoming alpha feature on a cluster before the feature is introduced, which means that prior to enabling an alpha feature a cluster admin has to sweep all possible objects to ensure no dangerous annotations are set.

This occurred for init containers - podsecuritypolicy is applied at creation time, but because init containers are represented as annotations a malicious user can set an init container annotation before Kube 1.3 is rolled out, and as soon as the cluster is upgraded that container would be executed. This means that admins have to run a pre-upgrade migration to purge any dangerous / out of policy annotations.

For these reasons, I believe we should not be using annotations for alpha level fields internally, and probably not externally.

Alternative

Instead, I believe we should represent alpha level fields in our Go structs and clearly denote them as alpha (perhaps with an named prefix). Once the feature is out of alpha, we would leave the alpha level field in the external struct for at least 1 major version to allow upgraders to use that field. We would then drop the alpha field, which would clean the database of that value. Internal code would simply rename the field, and ignore the external field.

Alpha level fields would be denoted by struct tags that allowed them to show up in swagger. Once we dropped support for the field, swagger would no longer show them. We would have to have a "drop by default" rule in place for swagger validations on fields that are no longer recognized for old alpha fields that aren't supported, but that's what we want for alpha.

E.g.:

Kube 1.2:

pod:
spec:
alphaInitContainers: # silently dropped by the storage layer
- name: init-1

Kube 1.3:

pod:
spec:
alphaInitContainers: # visible in swagger
- name: init-1
...

$ kubectl explain pod.spec.alphaInitContainers
This field allows you to provide specify "init containers" - run before other containers.
...
Feature is alpha and may change without notice.

Kube 1.4:

pod:
spec:
alphaInitContainers: # ignored, but visible in swagger. kubectl validate could flag this as unsupported
- name: init-1
initContainers:
- name: init-1

$ kubectl explain pod.spec.alphaInitContainers
This field has been removed - use pod.spec.initContainers

Kube 1.5

pod:
spec:
alphaInitContainers: # fails kubectl validate - alpha is no longer supported
# silently dropped from storage whenever a pod is updated
initContainers:
- name: init-1

$ kubectl explain pod.spec.alphaInitContainers
This field has been removed - use pod.spec.initContainers

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): Kind of. More like a bug report.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes apiserver CPU usage 100%

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.3

Environment:

Cloud provider or hardware configuration: AWS

OS (e.g. from /etc/os-release): CoreOS 1122.3.0

Kernel (e.g. uname -a):4.7.0-coreos-r1

Install tools: kube-aws

Others:

What happened: /hyperkube apiserver on the controller takes up all the CPU time, together with the kernel.

What you expected to happen: /hyperkube apiserver doesn't use up all the CPU time.

How to reproduce it (as minimally and precisely as possible): Run the cluster for a long time.

Anything else do we need to know:

The problem did not exist in the beginning. I started noticing it today. Our performance logs are only kept back for about 2 weeks. The controller shows the high CPU usage for that whole time frame.

Log from the kubelet unit:

[continues infinitely]
Nov 11 18:11:03 ip-10-0-0-50.eu-west-1.compute.internal kubelet-wrapper[1501]: I1111 18:11:03.940785 1501 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/db72bac5-8625-11e6-ad16-0253cca96ab9-default-token-6vjgz" (spec.Name: "default-token-6vjgz") pod "db72bac5-8625-11e6-ad16-0253cca96ab9" (UID: "db72bac5-8625-11e6-ad16-0253cca96ab9").
Nov 11 18:11:23 ip-10-0-0-50.eu-west-1.compute.internal kubelet-wrapper[1501]: I1111 18:11:23.735952 1501 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/secret/779b0278-9a1c-11e6-ad16-0253cca96ab9-default-token-6vjgz" (spec.Name: "default-token-6vjgz") to pod "779b0278-9a1c-11e6-ad16-0253cca96ab9" (UID: "779b0278-9a1c-11e6-ad16-0253cca96ab9"). Volume is already mounted to pod, but remount was requested.
Nov 11 18:11:23 ip-10-0-0-50.eu-west-1.compute.internal kubelet-wrapper[1501]: I1111 18:11:23.736264 1501 reconciler.go:254] MountVolume operation started for volume "kubernetes.io/configmap/779b0278-9a1c-11e6-ad16-0253cca96ab9-td-agent-config" (spec.Name: "td-agent-config") to pod "779b0278-9a1c-11e6-ad16-0253cca96ab9" (UID: "779b0278-9a1c-11e6-ad16-0253cca96ab9"). Volume is already mounted to pod, but remount was requested.
Nov 11 18:11:23 ip-10-0-0-50.eu-west-1.compute.internal kubelet-wrapper[1501]: I1111 18:11:23.744729 1501 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/configmap/779b0278-9a1c-11e6-ad16-0253cca96ab9-td-agent-config" (spec.Name: "td-agent-config") pod "779b0278-9a1c-11e6-ad16-0253cca96ab9" (UID: "779b0278-9a1c-11e6-ad16-0253cca96ab9").
Nov 11 18:11:23 ip-10-0-0-50.eu-west-1.compute.internal kubelet-wrapper[1501]: I1111 18:11:23.745843 1501 operation_executor.go:740] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/779b0278-9a1c-11e6-ad16-0253cca96ab9-default-token-6vjgz" (spec.Name: "default-token-6vjgz") pod "779b0278-9a1c-11e6-ad16-0253cca96ab9" (UID: "779b0278-9a1c-11e6-ad16-0253cca96ab9").

# uptime
18:12:36 up 17:38, 1 user, load average: 2.14, 2.39, 2.35