Change:
What happened: my kubelet restart unexpectedly,i find log info has more "Failed to list" info and has fatal error: go of nil func value,and then kubelet restart Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"elasticsearch-token-99wc2": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] failed 5 attempts to update node lease, will retry after 10s Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"default-token-6wh7n": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-rules": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"service-registry": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:30 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] Failed to list v1beta1.RuntimeClass: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] object-"logging"/"telegraf-elasticsearch": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:32 mgt15 kubelet[6914]: fatal error: go of nil func value Feb 02 17:46:32 mgt15 kubelet[6914]: goroutine 357 [running]: Feb 02 17:46:32 mgt15 kubelet[6914]: (closableConn).Close(0x4002193f00, 0x0, 0x0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Dialer).CloseAll(0x4000bbaea0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).updateNodeStatus(0x40010ce000, 0x54d0978, 0x40010ce648) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- +0xf0 Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).syncNodeStatus(0x40010ce000) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- What you expected to happen: kubelet do not restart How to reproduce it (as minimally and precisely as possible): when have more info of "Failed to list" Environment: Kubernetes version (use ):v1.14.3 OS (e.g: ):Ubuntu 18.04.3 LTS Kernel (e.g. ):4.15.0-55-generic Network plugin and version (if this is a network-related bug):calico3.8.2
/sig node /wg node /committee node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node /wg node /committee node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix is in
/sig auth
/assign
thankswhat version is this bug been fixed?
The fix is not yet merged, but I plan to backport the fix to 1.15.x, 1.16.x, 1.17.x (the exact patch versions depend on when the backports are merged)

Discussion:
closable := &closableConn{Conn: conn} <del> // Start tracking the connection d.mu.Lock() d.conns[closable] = struct{}{} d.mu.Unlock() </del> // When the connection is closed, remove it from the map. This will // be no-op if the connection isn't in the map, e.g. if CloseAll() // is called.

d.mu.Unlock() } <ins> // Start tracking the connection d.mu.Lock() d.conns[closable] = struct{}{} d.mu.Unlock() </ins> return closable, nil }

import ( "context" "net" <ins> "sync" "sync/atomic" </ins> "testing" "time" )

} } <ins> // TestCloseAllRace ensures CloseAll works with connections being simultaneously dialed func TestCloseAllRace(t *testing.T) { conns := int64(0) dialer := NewDialer(func(ctx context.Context, network, address string) (net.Conn, error) { return closeOnlyConn{onClose: func() { atomic.AddInt64(&conns, -1) }}, nil }) done := make(chan struct{}) wg := &sync.WaitGroup{} // Close all as fast as we can wg.Add(1) go func() { defer wg.Done() for { select { case <-done: return default: dialer.CloseAll() } } }() // Dial as fast as we can wg.Add(1) go func() { defer wg.Done() for { select { case <-done: return default: if _, err := dialer.Dial("", ""); err != nil { t.Error(err) return } atomic.AddInt64(&conns, 1) } } }() // Soak to ensure no races time.Sleep(time.Second) // Signal completion close(done) // Wait for goroutines wg.Wait() // Ensure CloseAll ran after all dials dialer.CloseAll() // Expect all connections to close within 5 seconds for start := time.Now(); time.Now().Sub(start) < 5*time.Second; time.Sleep(10 * time.Millisecond) { // Ensure all connections were closed if c := atomic.LoadInt64(&conns); c == 0 { break } else { t.Logf("got %d open connections, want 0, will retry", c) } } // Ensure all connections were closed if c := atomic.LoadInt64(&conns); c != 0 { t.Fatalf("got %d open connections, want 0", c) } } </ins> type closeOnlyConn struct { net.Conn onClose func()