Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all. I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix? :)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via

Discussion:
go_test( name = "go_default_test", <del> srcs = ["defaults_test.go"], </del> <ins> srcs = [ "conversion_test.go", "defaults_test.go", ], </ins> embed = [":go_default_library"], deps = [ "//pkg/api/legacyscheme:go_default_library",

"//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", <ins> "//vendor/github.com/stretchr/testify/assert:go_default_library", </ins> "//vendor/k8s.io/utils/pointer:go_default_library", ], )

} func Convert_autoscaling_PodsMetricSource_To_v2beta1_PodsMetricSource(in *autoscaling.PodsMetricSource, out *autoscalingv2beta1.PodsMetricSource, s conversion.Scope) error { <del> targetAverageValue := *in.Target.AverageValue out.TargetAverageValue = targetAverageValue </del> <ins> if in.Target.AverageValue != nil { targetAverageValue := *in.Target.AverageValue out.TargetAverageValue = targetAverageValue } </ins> out.MetricName = in.Metric.Name out.Selector = in.Metric.Selector

} out.MetricName = in.Metric.Name out.Selector = in.Metric.Selector <del> currentAverageValue := *in.Current.AverageValue out.AverageValue = &currentAverageValue </del> <ins> if in.Current.AverageValue != nil { currentAverageValue := *in.Current.AverageValue out.AverageValue = &currentAverageValue } </ins> return nil }

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v2beta1 import ( "testing" "github.com/stretchr/testify/assert" "k8s.io/api/autoscaling/v2beta1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/kubernetes/pkg/apis/autoscaling" ) // Testing nil pointer panic uncovered by #70806 // TODO(yue9944882): Test nil/empty conversion across all resource types func TestNilOrEmptyConversion(t *testing.T) { scheme := runtime.NewScheme() assert.NoError(t, addConversionFuncs(scheme)) testCases := []struct { obj1 interface{} obj2 interface{} }{ { obj1: &autoscaling.ExternalMetricSource{}, obj2: &v2beta1.ExternalMetricSource{}, }, { obj1: &autoscaling.ExternalMetricStatus{}, obj2: &v2beta1.ExternalMetricStatus{}, }, { obj1: &autoscaling.PodsMetricSource{}, obj2: &v2beta1.PodsMetricSource{}, }, { obj1: &autoscaling.PodsMetricStatus{}, obj2: &v2beta1.PodsMetricStatus{}, }, { obj1: &autoscaling.ObjectMetricSource{}, obj2: &v2beta1.ObjectMetricSource{}, }, { obj1: &autoscaling.ObjectMetricStatus{}, obj2: &v2beta1.ObjectMetricStatus{}, }, { obj1: &autoscaling.ResourceMetricSource{}, obj2: &v2beta1.ResourceMetricSource{}, }, { obj1: &autoscaling.ResourceMetricStatus{}, obj2: &v2beta1.ResourceMetricStatus{}, }, { obj1: &autoscaling.HorizontalPodAutoscaler{}, obj2: &v2beta1.HorizontalPodAutoscaler{}, }, { obj1: &autoscaling.MetricTarget{}, obj2: &v2beta1.CrossVersionObjectReference{}, }, } for _, testCase := range testCases { assert.NoError(t, scheme.Convert(testCase.obj1, testCase.obj2, nil)) assert.NoError(t, scheme.Convert(testCase.obj2, testCase.obj1, nil)) } } </ins>