Change:
return grpc.Dial( string(addr), <ins> grpc.WithAuthority("localhost"), </ins> grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target)

Discussion:
When registering a new driver the client is sending the filepath of the socket as the / header instead of . In practice almost all / servers outright reject this as invalid. Given that the client was updated years ago I'm unclear why this problem still persists to this day :( kubelet logs look like this: csi server logs look like this: Registration to move along happily. simple instructions here: Most drivers are written in and the server simply ignores this non-compliant header, otherwise the issue would be far more prevalent. <detailsEvery version up until current AFAIK. </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
I ran into this error in my use-case and debugged what's going on. I think the problem goes away with a recent enough and when the socket path is correctly prefixed in the client connection: I though I could start working on fixing this in kubelet.
That would be awesome! Then I can finally do away with all the crazy workarounds.
The nomad issue link you gave tried the approach I was talking about without luck it seems. I need take a closer look what's wrong...
btw, this look a duplicate of and
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale currently planning to use but I have some cleanup pending with first
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale The fix PR is pending approval

When using the CSI driver node registration sidecar container, the kubelet-registration-path parameter is set to either or . One or both of these options should cause kubelet to send a valid authority header to the socket. In the former case, kubelet will fail to find the file path since it will pass the unix header in the net.Dialer target to dial. In the latter, the dialer will succeed to call the container over the socket but send an incorrect :authority pseudo header (the ). A call into the CSI container with a valid authority header, using one or either kubelet-registration-path parameter. Deploy a CSI plugin example, setting kubelet-registration-path on the node-driver-registrar sidecar container to either a or . Note, a plugin may not fail in the latter case if the CSI plugin is written in a language with an http2 library that does not strictly check the authority header, however the header will still be incorrect. If using Rust as a language, the h2 library will strictly check the authority header and return a protocol error. Go does not seem to reject the invalid authority header, which is perhaps why most plugins do not notice the issue. Using will not have a 'unix:' header. It checks for this to substitute 'localhost' as the authority , which will not happen in this case. Using will get the authority substituted, but will pass the full 'unix:' header in as part of the path file. Related code can be seen (non-nil custom dialer) and (newGrpcConn looks like it expects no unix header based on log entry and externally supplied dialcontext). This work may have been overlapping with related work in grpc-go and plans , where both libraries seem to be taking responsibility for managing the authority header for unix sockets now. Kubelet logs using the unix header: Of note: the inconsistent 'Error while dialing dial unix' entries: one showing the unix: header for the path, while the registration socket shows it without. Also, the 'localhost' replacement is visible in the logs for the first (unix:-prefixed) and not the second registration notification call (non-prefixed). node registration sidecar logs when not_ using the unix header: Of note, kubelet notifies the container it received a protocol error.

The CSI container rust logs show the matching protocol error and authority header value: <details</details<detailsBare metal </details<details</details<details</details<detailscri-o </details<detailscalico, metal-lb, bgp, applicable 1.23 versions. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

ref: Target should start with unix: to avoid setting empty authority header according to RFC-3986.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
looks same as and
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details