Change:
Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] Expected error: <_errors.errorString | 0xc208641770: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"n [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}:nCommand stdout:nnstderr:nerror: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"nn", } Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}: Command stdout: stderr: error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" not to have occurred* See and Investigating... cc
Okiedokie. Looks like the v1.1 client is getting a response from the v1.2 master, but the v1.1 client doesn't know how to handle the v1 APIVersions object, because in v1.1, that object was never expected to have an . From a 1.1 server, we get which doesn't include a . We should expect to have plenty of pre-v1.1.7 clients floating around, and releasing a fixed v1.1.8 probably isn't going to solve this; if we want to be backward compatible, we need to not return an string with unversioned kinds like . Looking into more details about how we got here... cc cc
I have a maybe feeling that this is caused by ? In particular, we went from: to
Probably it's caused by this:
Let me double check the code. cc
What about that line? I've never dealt with the version negotiation; should the server and client be negotiating in a way that the server can return an object without ?
More precisely it's this line: This line sets the version to "v1". In , during encoding, we set unversioned object's APIVersion field to "v1".
I think in the long run, because we expect to move many unversioned types to group "common", so they should be versioned, so making APIVersion=v1 is the right move. Back to the backwards compatibility issue, how's the situation at other url, like http://127.0.0.1:8080/apis/extensions? That address also returns a versioned object, which at release 1.1 is unversioned: Do we need to keep all unversioned objects unversioned, or do we only need to ensure the server returns unversioned object at api/?
cc
Per our versioning rules, we should expect a v1.2 server to remain backward compatible with v1.1 clients. I don't have much experience with the api machinery, but if clients weren't expecting an unversioned object, it seems like we can't up and switch to versioned objects.
I think we have to maintain compatibility somehow. any ideas for a fix other than the obvious one?
Yes, we have to maintain backward compatibility with 1.1 (and 1.0) clients. New paths that old clients didn't know about could behave differently than paths that existed in previous releases.
Nope, let's make sure we leave a test and comment.
We can't make the change here though, this would break "versioned unversioned" objects (like )
The other implication of course is now we "versioned unversioned" objects (v1 status) and "unversioned unversioned", which means we shouldn't ever add new objects that don't return a version. I don't think anything in 1.2 qualifies as a new object without a version, just 1.1 objects that didn't have correct versions. Having objects from extensions group be returned without versions terrifies me - I don't see any way of describing that except as a bug.
We may need per API group version rules registered on scheme about whether "unversioned" objects are returned in their scheme version or as empty. That's probably the backward compatible fix that gives us a hope of reasoning about behavior correctly (status in 1.0 is v1, status in extensions might be "", apiversions in any new group is "v1"). The unversioned marker on the objects should ensure 1.2 clients can receive objects in both "" and "v1" form, we should add a test to verify that. I'm against letting empty API versions spread, but agree backcompat for existing groups versions should match previous behavior.
Can we make (or use) a different serializer/codec just for the endpoints that 1.1 clients used?
Probably we just need to change the version registered here:
That won't work like you expect - the type registry then has no knowledge in v1 of what schema the object is. That's not what we want - we want to not serialize the version of the object for specific API groups, which is a much more targeted change. This is a special, exceptional case (v1 group should return v1 Status. v1beta1 should not). , Chao Xu wrote:
Changing v1 breaks 1.0 backwards compat (we always returned apiVersion=v1 for 1.0) , Clayton Coleman wrote:
I mean this: Will this work?
I'm not sure without looking at the outcome (which is why unversioned is a bit of a disaster in our codebase). I would much prefer to explicitly identify which objects should have an unset apiversion via a method on the scheme, and do the explicit check at the time we do convert to version. That's an extremely controlled change, is obvious, and doesn't make the unversioned problem worse.
I remember somewhere in asserts version cannot be empty, so I guess my proposed solution won't work without extra work. I'll take the approach suggested by

Discussion:
func AddApiWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, getAPIVersionsFunc func(req *restful.Request) *unversioned.APIVersions) { // TODO: InstallREST should register each version automatically <del> versionHandler := APIVersionHandler(s, getAPIVersionsFunc) </del> <ins> // Because in release 1.1, /api returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} versionHandler := APIVersionHandler(ss, getAPIVersionsFunc) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

container.Add(ws) } <ins> // stripVersionEncoder strips APIVersion field from the encoding output. It's // used to keep the responses at the discovery endpoints backward compatible // with release-1.1, when the responses have empty APIVersion. type stripVersionEncoder struct { encoder    runtime.Encoder serializer runtime.Serializer } func (c stripVersionEncoder) EncodeToStream(obj runtime.Object, w io.Writer, overrides ...unversioned.GroupVersion) error { buf := bytes.NewBuffer([]byte{}) err := c.encoder.EncodeToStream(obj, buf, overrides...) if err != nil { return err } roundTrippedObj, gvk, err := c.serializer.Decode(buf.Bytes(), nil, nil) if err != nil { return err } gvk.Group = "" gvk.Version = "" roundTrippedObj.GetObjectKind().SetGroupVersionKind(gvk) return c.serializer.EncodeToStream(roundTrippedObj, w) } // StripVersionNegotiatedSerializer will return stripVersionEncoder when // EncoderForVersion is called. See comments for stripVersionEncoder. type StripVersionNegotiatedSerializer struct { runtime.NegotiatedSerializer } func (n StripVersionNegotiatedSerializer) EncoderForVersion(serializer runtime.Serializer, gv unversioned.GroupVersion) runtime.Encoder { encoder := n.NegotiatedSerializer.EncoderForVersion(serializer, gv) return stripVersionEncoder{encoder, serializer} } func keepUnversioned(group string) bool { return group == "" || group == "extensions" } </ins> // Adds a service to return the supported api versions at /apis. func AddApisWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) { <del> rootAPIHandler := RootAPIHandler(s, f) </del> <ins> // Because in release 1.1, /apis returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} rootAPIHandler := RootAPIHandler(ss, f) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

// Adds a service to return the supported versions, preferred version, and name // of a group. E.g., a such web service will be registered at /apis/extensions. func AddGroupWebService(s runtime.NegotiatedSerializer, container *restful.Container, path string, group unversioned.APIGroup) { <del> groupHandler := GroupHandler(s, group) </del> <ins> ss := s if keepUnversioned(group.Name) { // Because in release 1.1, /apis/extensions returns response with empty // APIVersion, we use StripVersionNegotiatedSerializer to keep the // response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } groupHandler := GroupHandler(ss, group) </ins> ws := new(restful.WebService) ws.Path(path) ws.Doc("get information of a group")

// Adds a service to return the supported resources, E.g., a such web service // will be registered at /apis/extensions/v1. func AddSupportedResourcesWebService(s runtime.NegotiatedSerializer, ws *restful.WebService, groupVersion unversioned.GroupVersion, apiResources []unversioned.APIResource) { <del> resourceHandler := SupportedResourcesHandler(s, groupVersion, apiResources) </del> <ins> ss := s if keepUnversioned(groupVersion.Group) { // Because in release 1.1, /apis/extensions/v1beta1 returns response // with empty APIVersion, we use StripVersionNegotiatedSerializer to // keep the response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } resourceHandler := SupportedResourcesHandler(ss, groupVersion, apiResources) </ins> ws.Route(ws.GET("/").To(resourceHandler). Doc("get available resources"). Operation("getAPIResources").

assert.Equal([]string{"127.0.0.2", "127.0.0.2"}, addrs) } <ins> // Because we need to be backwards compatible with release 1.1, at endpoints // that exist in release 1.1, the responses should have empty APIVersion. func TestAPIVersionOfDiscoveryEndpoints(t *testing.T) { master, etcdserver, _, assert := newMaster(t) defer etcdserver.Terminate(t) server := httptest.NewServer(master.HandlerContainer.ServeMux) // /api exists in release-1.1 resp, err := http.Get(server.URL + "/api") if err != nil { t.Errorf("unexpected error: %v", err) } apiVersions := unversioned.APIVersions{} assert.NoError(decodeResponse(resp, &apiVersions)) assert.Equal(apiVersions.APIVersion, "") // /api/v1 exists in release-1.1 resp, err = http.Get(server.URL + "/api/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList := unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis exists in release-1.1 resp, err = http.Get(server.URL + "/apis") if err != nil { t.Errorf("unexpected error: %v", err) } groupList := unversioned.APIGroupList{} assert.NoError(decodeResponse(resp, &groupList)) assert.Equal(groupList.APIVersion, "") // /apis/extensions exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions") if err != nil { t.Errorf("unexpected error: %v", err) } group := unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "") // /apis/extensions/v1beta1 exists in release-1.1 resp, err = http.Get(server.URL + "/apis/extensions/v1beta1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "") // /apis/autoscaling doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling") if err != nil { t.Errorf("unexpected error: %v", err) } group = unversioned.APIGroup{} assert.NoError(decodeResponse(resp, &group)) assert.Equal(group.APIVersion, "v1") // apis/autoscaling/v1 doesn't exist in release-1.1, so the APIVersion field // should be non-empty in the results returned by the server. resp, err = http.Get(server.URL + "/apis/autoscaling/v1") if err != nil { t.Errorf("unexpected error: %v", err) } resourceList = unversioned.APIResourceList{} assert.NoError(decodeResponse(resp, &resourceList)) assert.Equal(resourceList.APIVersion, "v1") } </ins> func TestDiscoveryAtAPIS(t *testing.T) { master, etcdserver, config, assert := newMaster(t) defer etcdserver.Terminate(t)