Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.
Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes

Discussion:
d := schemaCoercingDecoder{delegate: ret.StorageConfig.Codec, validator: unstructuredSchemaCoercer{ // drop invalid fields while decoding old CRs (before we haven't had any ObjectMeta validation) dropInvalidMetadata:   true, <ins> repairGeneration:      true, </ins> structuralSchemas:     t.structuralSchemas, structuralSchemaGK:    t.structuralSchemaGK, preserveUnknownFields: t.preserveUnknownFields,

// - generic pruning of unknown fields following a structural schema. type unstructuredSchemaCoercer struct { dropInvalidMetadata bool <ins> repairGeneration    bool </ins> structuralSchemas     map[string]*structuralschema.Structural structuralSchemaGK    schema.GroupKind

if err := schemaobjectmeta.Coerce(nil, u.Object, v.structuralSchemas[gv.Version], false, v.dropInvalidMetadata); err != nil { return err } <ins> // fixup missing generation in very old CRs if v.repairGeneration && objectMeta.Generation == 0 { objectMeta.Generation = 1 } </ins> } // restore meta fields, starting clean

"apiVersion": "v1", "metadata": map[string]interface{}{ "generateName": nil, <ins> "generation":   nil, </ins> "labels": map[string]interface{}{ "foo": nil, },

t.Fatal(err) } if o.GenerateName != "" { <del> t.Errorf("expected null json value to be read as "" string, but got: %q", o.GenerateName) </del> <ins> t.Errorf("expected null json generateName value to be read as "" string, but got: %q", o.GenerateName) } if o.Generation != 0 { t.Errorf("expected null json generation value to be read as zero, but got: %q", o.Generation) </ins> } if got, expected := o.Labels, map[string]string{"foo": ""}; !reflect.DeepEqual(got, expected) { t.Errorf("unexpected labels, expected=%#v, got=%#v", expected, got)

if got, expected := o.GenerateName, pod.ObjectMeta.GenerateName; got != expected { t.Errorf("expected generatedName to be %q, got %q", expected, got) } <ins> if got, expected := o.Generation, pod.ObjectMeta.Generation; got != expected { t.Errorf("expected generation to be %q, got %q", expected, got) } </ins> if got, expected := o.Labels, pod.ObjectMeta.Labels; !reflect.DeepEqual(got, expected) { t.Errorf("expected labels to be %v, got %v", expected, got) }

obj := fixtures.NewNoxuInstance("default", "foo") unstructured.SetNestedField(obj.UnstructuredContent(), int64(42), "metadata", "unknown") <ins> unstructured.SetNestedField(obj.UnstructuredContent(), nil, "metadata", "generation") </ins> unstructured.SetNestedField(obj.UnstructuredContent(), map[string]interface{}{"foo": int64(42), "bar": "abc"}, "metadata", "labels") _, err = instantiateCustomResource(t, obj, noxuResourceClient, noxuDefinition) if err == nil {

} else if found { t.Errorf("unexpected metadata.unknown=%#v: expected this to be pruned", unknown) } <ins> if generation, found, err := unstructured.NestedInt64(obj.UnstructuredContent(), "metadata", "generation"); err != nil { t.Errorf("unexpected error getting metadata.generation: %v", err) } else if !found { t.Errorf("expected metadata.generation=1: got: %d", generation) } else if generation != 1 { t.Errorf("unexpected metadata.generation=%d: expected this to be set to 1", generation) } </ins> } func TestInvalidObjectMetaInStorage(t *testing.T) {

original := fixtures.NewNoxuInstance("default", "foo") unstructured.SetNestedField(original.UnstructuredContent(), int64(42), "metadata", "unknown") <ins> unstructured.SetNestedField(original.UnstructuredContent(), nil, "metadata", "generation") </ins> unstructured.SetNestedField(original.UnstructuredContent(), map[string]interface{}{"foo": int64(42), "bar": "abc"}, "metadata", "annotations") unstructured.SetNestedField(original.UnstructuredContent(), map[string]interface{}{"invalid": "x y"}, "metadata", "labels") unstructured.SetNestedField(original.UnstructuredContent(), int64(42), "embedded", "metadata", "unknown")

t.Errorf("Unexpected to find embedded.metadata.unknown=%#v", unknown) } <ins> t.Logf("Checking that metadata.generation=1") if generation, found, err := unstructured.NestedInt64(obj.UnstructuredContent(), "metadata", "generation"); err != nil { t.Errorf("unexpected error getting metadata.generation: %v", err) } else if !found { t.Errorf("expected metadata.generation=1: got: %d", generation) } else if generation != 1 { t.Errorf("unexpected metadata.generation=%d: expected this to be set to 1", generation) } </ins> t.Logf("Checking that ObjectMeta is pruned from wrongly-typed annotations") if annotations, found, err := unstructured.NestedStringMap(obj.UnstructuredContent(), "metadata", "annotations"); err != nil {