Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver is always restarting and I can't access it with any commands looks like . When I check the kubectl's status, the tells me that: And then I checked the container's status and log of the apiserver: status: log: I see that there is a line: And then I get the source code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.3. But I can't use the now because of the problem, so I can't give you a formal result about the version. Cloud provider or hardware configuration: Ryzen 5 4500U 16GB memory 30GB SSD In the VMware Workstation 15.5.6 OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
The complete version for the log of apiserver:
/cc
: did you tweak that machine's clock backwards, or run any software that does this?
Time calculations should be written to use go's , not count on wall clock time always moving foward.
: The proximal problem is that uses and gets the answer that time went backwards.
sure, but should be immune to wall clock resets per
Exactly. It looks to me like this test for ordering is using the monotonic clock and somehow that is saying time went backwards.
does the quantization () factor into well-ordered detection? I'm not sure how the monotonic clock interacts with Unix / UnixNano
No, I see the quantization is buggy, it uses wall clock. But the ordering test does not depend on the quantization.
/triage accepted /priority urgent-critical
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accepted /priority urgent-critical </details
very odd that this is repeatable. /priority critical-urgent
: it is very odd that this is repeatable. Do you have anything unusual in your environment ? I see it is vmware workstation. Does that mess with the clock in any unusual way? I also wonder about the golang runtime; did you do your own build (if so, what version of golang) or fetch binaries from somewhere (where?); full output would be helpful.
I don't have anything unusual in my environment. I run it in the ubuntu driven by vmware workstation, because my real machine's operation system is Windows10 , and I need a virtual machine to run k8s. My computer is a laptop computer. I got the golang runtime by apt-get. I also got the distribution of k8s by apt-get too. I did not do my own build. ! Okay, I got my golang and k8s version now: !
I have exactly same problem in my local lab. Virt infra on Virtual Box 6.1.16 Guest OS
you try and, if so, does it fix your problem and does it log complaints about the monotic clock going backwards? you tell us about your host and the settings on your VM(s)?
exactly matches same error as posted by author of issue, I have NOT tried VM settings: 4 vCPUs 4GB RAM swap is OFF on OS level. Host HW: Core i9 9900K 32GB Ram BTW: I have managed to install 1.18.12 on same VM eventually - cause for me it is just lab infra to study, so 1.19.X is NOT that important for me to have on this infra.
: host OS? Is your VB "Version 6.1.16 r140961 (Qt5.6.3)"? VM settings match the following? <img width="616" alt="Screen Shot 2020-11-16 at 4 56 10 PM" src="https://user-"<img width="742" alt="Screen Shot 2020-11-16 at 4 55 58 PM" src="https://user-"<img width="720" alt="Screen Shot 2020-11-16 at 4 55 45 PM" src="https://user-"
OS Name: Microsoft Windows 10 Enterprise OS Version: 10.0. N/A Build OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free ! ! ! !
I tried to reproduce the problem using Ubuntu 18.0.4.5 and Kubernetes 1.19.4 (installed by kubeadm, using Calico as the network plugin), but the problem did not strike. The guest was created from the Ubuntu live Server ISO with a very plain install, picking only OpenSSH from the things to add during the install. No explicit tweaks regarding timekeeping. In the guest: I am also running the exact same version of VBox as reported above. I have the ExtensionPack of the same version installed. My host is Windows 10 Pro version 2004. Intel Core i9-9900 (8 hyperthreaded cores, thus 16 logical processors), 32 GB RAM. Configured for automatic time sync. Last sync 34 minutes ago, with server
Same issue in my env: VM (CentOS 7.9.2009 4 CPU+12G Memory) on virtualbox(6.1) Host is Windows 10 (16G Memory, CPU Intel(R) Core(TM) i3- CPU @ 3.60GHz 3.60 GHz) docker version: docker-ce-19.03.9-3.el7.x8664 K8S version: 1.19.4 Part error log output from docker logs -f <kube-apiserver-containerid:
Hi, same here vitualbox 6.1.18 latest Linux master-1 5.4.0-72-generic latest /etc/os-release NAME="Ubuntu" VERSION="20.04.2 LTS (Focal Fossa)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 20.04.2 LTS" VERSIONID="20.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=focal UBUNTU_CODENAME=focal tried with K8s 1st with 1.21.0-00 then 1.19.10-00 same Time went backwards message in the logs. Api server uses large amount of CPU at doing nothing. etcd cluster seems also to be totally lost. 3 masters setup. thanks
Hm ... this must have something to do with the fact we're running in a VM. Funny enough ... you guys are reporting apiserver crashing. What led me to this issue is that my metrics-server crashes with this same log message instead :). I originally had ntpd installed on the VM, but removing it did not help. So ... if time goes backward, it must be something that VirtualBox does. I'll try to see if I can repro with HyperV. EDIT: I was able to get rid of this error with VirtualBox by using to disable some kind of automated clock sync that VirtualBox does by default. Don't forget to use NTPd, as you will need some time synchronization :). EDIT 2: Spoke too soon. That did not really help. It seems to have slightly delayed the issue, though, as it only presented itself after a couple of minutes... EDIT 3: So, the reason why it's still syncing time, is VBoxGuestAdditions. When they are running, the guest synces time with the host, even with the above time-sync setting disabled. One either has to stop the guest additions or pass them (but ... it's not really a configuration change: ). EDIT 4: The above doesn't seem to have an effect at all. Still, according to the metrics server, time goes backwards. Well ... let's try to completely disable the VirtualBox guest additions: (for my use case, this is totally fine, as I don't need any of the guest additions features such as shared folders or time sync). EDIT 5: Nothing helps. I reckon, either the check for "time went backwards" is somewhat buggy, or some odd time sync still takes place. For now, because I don't really need it, I am just disabling the metrics server. EDIT 6: Huh. Got it, finally. Never surrender, I guess :). Turns out, you gotta attack it from all sides. (see above) Add the argument to (see above) (this is new; turns out, even if one tells guest additions not to sync time, and disables the time syncing on the VM, something can still sync the time -- this setting makes the guest OS completely blind to the fact it is running in a VM). I suggest to keep ntp and guest additions (with the disabled time sync) on at all times. EDIT 7: Also ... do no let your host sleep. When it sleeps and then wakes up, a lot of k8s cannot cope with it and begins rolling. It should recover, but if you can avoid sleeping, do. For me, this actually meant installing and using "Don't sleep" (alternative to "Caffeine" or similar) to keep it awake. For some reason, when you lock your computer, Windows will try to sleep. It should be possible to disable this mechanic (), but even though I did, it would still sleep. That's why I ended up with "Don't sleep". With all of this in place, it finally works as expected. None of the k8s' pods roll anymore.
I had exactly the same issue. Followed the steps listed above and it solved my issue. Thanks , much appreciated!

Discussion:
// SampleAndWaterMarkObserverGenerator creates TimedObservers that // populate histograms of samples and low- and high-water-marks.  The // generator has a samplePeriod, and the histograms get an observation <del> // every samplePeriod. </del> <ins> // every samplePeriod.  The sampling windows are quantized based on // the monotonic rather than wall-clock times.  The `t0` field is // there so to provide a baseline for monotonic clock differences. </ins> type SampleAndWaterMarkObserverGenerator struct { *sampleAndWaterMarkObserverGenerator } type sampleAndWaterMarkObserverGenerator struct { clock        clock.PassiveClock <ins> t0           time.Time </ins> samplePeriod time.Duration samples      *compbasemetrics.HistogramVec waterMarks   *compbasemetrics.HistogramVec

return SampleAndWaterMarkObserverGenerator{ &sampleAndWaterMarkObserverGenerator{ clock:        clock, <ins> t0:           clock.Now(), </ins> samplePeriod: samplePeriod, samples:      compbasemetrics.NewHistogramVec(sampleOpts, labelNames), waterMarks:   compbasemetrics.NewHistogramVec(waterMarkOpts, append([]string{labelNameMark}, labelNames...)),

} func (swg *sampleAndWaterMarkObserverGenerator) quantize(when time.Time) int64 { <del> return when.UnixNano() / int64(swg.samplePeriod) </del> <ins> return int64(when.Sub(swg.t0) / swg.samplePeriod) </ins> } // Generate makes a new TimedObserver

} func (saw *sampleAndWaterMarkHistograms) innerSet(updateXOrX1 func()) { <del> saw.Lock() when := saw.clock.Now() whenInt := saw.quantize(when) acc := saw.sampleAndWaterMarkAccumulator wellOrdered := !when.Before(acc.lastSet) if wellOrdered { </del> <ins> var when time.Time var whenInt int64 var acc sampleAndWaterMarkAccumulator var wellOrdered bool func() { saw.Lock() defer saw.Unlock() when = saw.clock.Now() whenInt = saw.quantize(when) acc = saw.sampleAndWaterMarkAccumulator wellOrdered = !when.Before(acc.lastSet) </ins> updateXOrX1() saw.relX = saw.x / saw.x1 <del> if acc.lastSetInt < whenInt { saw.loRelX, saw.hiRelX = acc.relX, acc.relX saw.lastSetInt = whenInt </del> <ins> if wellOrdered { if acc.lastSetInt < whenInt { saw.loRelX, saw.hiRelX = acc.relX, acc.relX saw.lastSetInt = whenInt } saw.lastSet = when </ins> } <ins> // `wellOrdered` should always be true because we are using // monotonic clock readings and they never go backwards.  Yet // very small backwards steps (under 1 microsecond) have been // observed // (https://github.com/kubernetes/kubernetes/issues/96459). // In the backwards case, treat the current reading as if it // had occurred at time `saw.lastSet` and log an error.  It // would be wrong to update `saw.lastSet` in this case because // that plants a time bomb for future updates to // `saw.lastSetInt`. </ins> if saw.relX < saw.loRelX { saw.loRelX = saw.relX } else if saw.relX > saw.hiRelX { saw.hiRelX = saw.relX } <del> saw.lastSet = when } saw.Unlock() </del> <ins> }() </ins> if !wellOrdered { <del> lastSetS := acc.lastSet.Format(time.RFC3339Nano) whenS := when.Format(time.RFC3339Nano) klog.Fatalf("Time went backwards from %s to %s for labelValues=%#+v", lastSetS, whenS, saw.labelValues) panic(append([]string{lastSetS, whenS}, saw.labelValues...)) </del> <ins> lastSetS := acc.lastSet.String() whenS := when.String() klog.Errorf("Time went backwards from %s to %s for labelValues=%#+v", lastSetS, whenS, saw.labelValues) </ins> } for acc.lastSetInt < whenInt { saw.samples.WithLabelValues(saw.labelValues...).Observe(acc.relX)