Change:
pull-kubernetes-unit, see example failure in It appears to be a new test in No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Repro's locally:
/cc can you PTAL? It seems related to new test in
flaking regularly -
actually, I'm not sure this is a test-only race... it looks like self mutates , and that is called in a goroutine in parallel with other methods that access rootDirectory: I suspect this has always been a race, and the unit test is exercising / catching it more
moving the filepath.Clean to init time: and changing the check in setupDataDirs to be a validation rather than a mutation: might resolve the immediate race
/assign
is open to fixing it. The data race we find here should be fixed. However, is another one that we need to fix. I add a new commit to fix that as well.
did you see that race as well or did you just see the issue opened? that was in a code path that should be feature gated so I wouldn't have expected it to trigger in the kubemark unit tests
I may mixed problems together. Yes. will not trigger in the kubemark unit tests. Using stressing, I find the data race cannot fix the problem completely. Then I saw and cherry-picked the fix to my PR. But the hollowNode test still flakes and I looked into it and find it is the volumeManager problem. is just another potential data race in .
the rootDirectory and volumeManagerSyncTime race fixes are easy to reason about and fix code that looks like it could really race in currently enabled-by-default code looks like a real race in disabled-by-default code. I'll defer to kubelet folks and whether to mix that fix into
I think I should drop that fix. This will make things more clear.

Discussion:
import ( "fmt" _ "net/http/pprof" // Enable pprof HTTP handlers. <ins> "path/filepath" </ins> "strings" "github.com/spf13/pflag"

return &KubeletFlags{ ContainerRuntimeOptions: *NewContainerRuntimeOptions(), CertDirectory:           "/var/lib/kubelet/pki", <del> RootDirectory:           defaultRootDir, </del> <ins> RootDirectory:           filepath.Clean(defaultRootDir), </ins> MaxContainerCount:       -1, MaxPerPodContainerCount: 1, MinimumGCAge:            metav1.Duration{Duration: 0},

kubeClient:                              kubeDeps.KubeClient, heartbeatClient:                         kubeDeps.HeartbeatClient, onRepeatedHeartbeatFailure:              kubeDeps.OnHeartbeatFailure, <del> rootDirectory:                           rootDirectory, </del> <ins> rootDirectory:                           filepath.Clean(rootDirectory), </ins> resyncInterval:                          kubeCfg.SyncFrequency.Duration, sourcesReady:                            config.NewSourcesReady(kubeDeps.PodConfig.SeenAllSources), registerNode:                            registerNode,

// 4.  the pod-resources directory // 5.  the checkpoint directory func (kl *Kubelet) setupDataDirs() error { <del> kl.rootDirectory = filepath.Clean(kl.rootDirectory) </del> <ins> if cleanedRoot := filepath.Clean(kl.rootDirectory); cleanedRoot != kl.rootDirectory { return fmt.Errorf("rootDirectory not in canonical form: expected %s, was %s", cleanedRoot, kl.rootDirectory) } </ins> pluginRegistrationDir := kl.getPluginsRegistrationDir() pluginsDir := kl.getPluginsDir() if err := os.MkdirAll(kl.getRootDir(), 0750); err != nil {

import ( "context" "os" <ins> "path/filepath" </ins> "testing" "time"

} defer os.RemoveAll(basePath) kb := &Kubelet{ <del> rootDirectory:    basePath, </del> <ins> rootDirectory:    filepath.Clean(basePath), </ins> recorder:         &record.FakeRecorder{}, cadvisor:         cadvisor, nodeLister:       testNodeLister{},

import ( "fmt" <ins> "sync" </ins> "time" v1 "k8s.io/api/core/v1"

volumePluginMgr               *volumepkg.VolumePluginMgr skippedDuringReconstruction   map[v1.UniqueVolumeName]*globalVolumeInfo kubeletPodsDir                string <del> timeOfLastSync                time.Time volumesFailedReconstruction   []podVolume volumesNeedDevicePath         []v1.UniqueVolumeName volumesNeedReportedInUse      []v1.UniqueVolumeName </del> <ins> // lock protects timeOfLastSync for updating and checking timeOfLastSyncLock          sync.Mutex timeOfLastSync              time.Time volumesFailedReconstruction []podVolume volumesNeedDevicePath       []v1.UniqueVolumeName volumesNeedReportedInUse    []v1.UniqueVolumeName </ins> } func (rc *reconciler) Run(stopCh <-chan struct{}) {

} func (rc *reconciler) updateLastSyncTime() { <ins> rc.timeOfLastSyncLock.Lock() defer rc.timeOfLastSyncLock.Unlock() </ins> rc.timeOfLastSync = time.Now() } func (rc *reconciler) StatesHasBeenSynced() bool { <ins> rc.timeOfLastSyncLock.Lock() defer rc.timeOfLastSyncLock.Unlock() </ins> return !rc.timeOfLastSync.IsZero() }