Change:
What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it. [1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting

Discussion:
// ListenAndServePodResources runs the kubelet podresources grpc service func (kl *Kubelet) ListenAndServePodResources() { <del> server.ListenAndServePodResources(util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket), kl.podManager, kl.containerManager) </del> <ins> socket, err := util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket) if err != nil { klog.V(2).Infof("Failed to get local endpoint for PodResources endpoint: %v", err) return } server.ListenAndServePodResources(socket, kl.podManager, kl.containerManager) </ins> } // Delete the eligible dead container instances in a pod. Depending on the configuration, the latest dead containers may be kept around.

} // LocalEndpoint returns the full path to a unix socket at the given endpoint <del> func LocalEndpoint(path, file string) string { </del> <ins> func LocalEndpoint(path, file string) (string, error) { </ins> u := url.URL{ Scheme: unixProtocol, Path:   path, } <del> return filepath.Join(u.String(), file+".sock") </del> <ins> return filepath.Join(u.String(), file+".sock"), nil </ins> }

} // LocalEndpoint empty implementation <del> func LocalEndpoint(path, file string) string { return "" </del> <ins> func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } // GetBootTime empty implementation

} } <del> // LocalEndpoint returns the full path to a windows named pipe func LocalEndpoint(path, file string) string { u := url.URL{ Scheme: npipeProtocol, Path:   path, } return u.String() + "//./pipe/" + file </del> <ins> // LocalEndpoint empty implementation func LocalEndpoint(path, file string) (string, error) { return "", fmt.Errorf("LocalEndpoints are unsupported in this build") </ins> } var tickCount = syscall.NewLazyDLL("kernel32.dll").NewProc("GetTickCount64")

} func getNodeDevices() (*podresourcesapi.ListPodResourcesResponse, error) { <del> endpoint := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) </del> <ins> endpoint, err := util.LocalEndpoint(defaultPodResourcesPath, podresources.Socket) if err != nil { return nil, fmt.Errorf("Error getting local endpoint: %v", err) } </ins> client, conn, err := podresources.GetClient(endpoint, defaultPodResourcesTimeout, defaultPodResourcesMaxSize) if err != nil { return nil, fmt.Errorf("Error getting grpc client: %v", err)