Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work? I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2. I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/

Discussion:
github.com/urfave/negroni v1.0.0 // indirect github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 // indirect <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 // indirect

github.com/urfave/negroni => github.com/urfave/negroni v1.0.0 github.com/vishvananda/netlink => github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 <del> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk => github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack => github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xiang90/probing => github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18

github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk= github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 h1:J9gO8RJCAFlln1jsvRba/CWVUnMHwObklfxxjErl1uk= github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI= <del> github.com/vmware/govmomi v0.20.0 h1:+1IyhvoVb5JET2Wvgw9J3ZDv6CK4sxzUunpH8LhQqm4= github.com/vmware/govmomi v0.20.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc h1:uyYiBHCUWsifUdIWrBzUUq9B3FSZKFTAuUyMQczS1q8= github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc/go.mod h1:e6humHha1ekIwTCm+A5Qed5mG8V4JL+ChHcUOJ+L/8U= github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 h1:3OK6oXqbbEUeWvTJgg1pZgNjzdCNE6hMu2TragCfwb8=

github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= <del> github.com/vmware/govmomi v0.20.0 h1:+1IyhvoVb5JET2Wvgw9J3ZDv6CK4sxzUunpH8LhQqm4= github.com/vmware/govmomi v0.20.0/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 h1:a4tQYYYuK9QdeO/+kEvNYyuR21S+7ve5EANok6hABhI= golang.org/x/crypto v0.0.0-20181025213731-e84da0312774/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=

} bhash := sha256.New().Sum(body) <del> // Port in the signature must be that of the reverse proxy port, vCenter's default is port 80 port := "80" // TODO: get from lookup service </del> <ins> port := req.URL.Port() if port == "" { port = "80" // Default port for the "Host" header on the server side } </ins> var buf bytes.Buffer msg := []string{ nonce,

github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 github.com/vishvananda/netns <del> # github.com/vmware/govmomi v0.20.0 => github.com/vmware/govmomi v0.20.0 </del> <ins> # github.com/vmware/govmomi v0.20.1 => github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/govmomi github.com/vmware/govmomi/find github.com/vmware/govmomi/list