Change:
BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to. What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.

Discussion:
# Make sure it's gone kube::test::get_object_assert foos "{{range.items}}{{$id_field}}:{{end}}" '' kube::test::get_object_assert bars "{{range.items}}{{$id_field}}:{{end}}" '' <ins> # Test 'kubectl create' with namespace, and namespace cleanup. kubectl "${kube_flags[@]}" create namespace non-native-resources kubectl "${kube_flags[@]}" create -f hack/testdata/TPR/bar.yaml --namespace=non-native-resources kube::test::get_object_assert bars '{{len .items}}' '1' --namespace=non-native-resources kubectl "${kube_flags[@]}" delete namespace non-native-resources # Make sure objects go away. kube::test::wait_object_assert bars '{{len .items}}' '0' --namespace=non-native-resources # Make sure namespace goes away. local tries=0 while kubectl "${kube_flags[@]}" get namespace non-native-resources && [ ${tries} -lt 10 ]; do tries=$((tries+1)) sleep ${tries} done </ins> } run_recursive_resources_tests() {

return ret } <ins> // In addition to Unstructured objects (Custom Resources), we also may sometimes need to // decode unversioned Options objects, so we delegate to parameterScheme for such types. parameterScheme := runtime.NewScheme() parameterScheme.AddUnversionedTypes(schema.GroupVersion{Group: crd.Spec.Group, Version: crd.Spec.Version}, &metav1.ListOptions{}, &metav1.ExportOptions{}, &metav1.GetOptions{}, &metav1.DeleteOptions{}, ) parameterScheme.AddGeneratedDeepCopyFuncs(metav1.GetGeneratedDeepCopyFuncs()...) parameterCodec := runtime.NewParameterCodec(parameterScheme) </ins> kind := schema.GroupVersionKind{Group: crd.Spec.Group, Version: crd.Spec.Version, Kind: crd.Spec.Names.Kind} <ins> typer := unstructuredObjectTyper{ delegate:          parameterScheme, unstructuredTyper: discovery.NewUnstructuredObjectTyper(nil), } creator := unstructuredCreator{} </ins> storage := customresource.NewREST( schema.GroupResource{Group: crd.Spec.Group, Resource: crd.Spec.Names.Plural}, schema.GroupVersionKind{Group: crd.Spec.Group, Version: crd.Spec.Version, Kind: crd.Spec.Names.ListKind}, UnstructuredCopier{}, customresource.NewStrategy( <del> discovery.NewUnstructuredObjectTyper(nil), </del> <ins> typer, </ins> crd.Spec.Scope == apiextensions.NamespaceScoped, kind, ), r.restOptionsGetter, ) <del> parameterScheme := runtime.NewScheme() parameterScheme.AddUnversionedTypes(schema.GroupVersion{Group: crd.Spec.Group, Version: crd.Spec.Version}, &metav1.ListOptions{}, &metav1.ExportOptions{}, &metav1.GetOptions{}, &metav1.DeleteOptions{}, ) parameterScheme.AddGeneratedDeepCopyFuncs(metav1.GetGeneratedDeepCopyFuncs()...) parameterCodec := runtime.NewParameterCodec(parameterScheme) </del> selfLinkPrefix := "" switch crd.Spec.Scope { case apiextensions.ClusterScoped:

return ret }, <del> Serializer:     UnstructuredNegotiatedSerializer{}, </del> <ins> Serializer:     unstructuredNegotiatedSerializer{typer: typer, creator: creator}, </ins> ParameterCodec: parameterCodec, <del> Creater:         UnstructuredCreator{}, </del> <ins> Creater:         creator, </ins> Convertor:       unstructured.UnstructuredObjectConverter{}, <del> Defaulter:       UnstructuredDefaulter{}, </del> <ins> Defaulter:       unstructuredDefaulter{parameterScheme}, </ins> Copier:          UnstructuredCopier{}, <del> Typer:           discovery.NewUnstructuredObjectTyper(nil), </del> <ins> Typer:           typer, </ins> UnsafeConvertor: unstructured.UnstructuredObjectConverter{}, Resource:    schema.GroupVersionResource{Group: crd.Spec.Group, Version: crd.Spec.Version, Resource: crd.Spec.Names.Plural},

return ret } <del> type UnstructuredNegotiatedSerializer struct{} </del> <ins> type unstructuredNegotiatedSerializer struct { typer   runtime.ObjectTyper creator runtime.ObjectCreater } </ins> <del> func (s UnstructuredNegotiatedSerializer) SupportedMediaTypes() []runtime.SerializerInfo { </del> <ins> func (s unstructuredNegotiatedSerializer) SupportedMediaTypes() []runtime.SerializerInfo { </ins> return []runtime.SerializerInfo{ { MediaType:        "application/json", EncodesAsText:    true, <del> Serializer:       json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), false), PrettySerializer: json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), true), </del> <ins> Serializer:       json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, false), PrettySerializer: json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, true), </ins> StreamSerializer: &runtime.StreamSerializerInfo{ EncodesAsText: true, <del> Serializer:    json.NewSerializer(json.DefaultMetaFactory, UnstructuredCreator{}, discovery.NewUnstructuredObjectTyper(nil), false), </del> <ins> Serializer:    json.NewSerializer(json.DefaultMetaFactory, s.creator, s.typer, false), </ins> Framer:        json.Framer, }, }, } } <del> func (s UnstructuredNegotiatedSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { </del> <ins> func (s unstructuredNegotiatedSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { </ins> return versioning.NewDefaultingCodecForScheme(Scheme, crEncoderInstance, nil, gv, nil) } <del> func (s UnstructuredNegotiatedSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return unstructured.UnstructuredJSONScheme </del> <ins> func (s unstructuredNegotiatedSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return unstructuredDecoder{delegate: Codecs.DecoderToVersion(serializer, gv)} } type unstructuredDecoder struct { delegate runtime.Decoder } func (d unstructuredDecoder) Decode(data []byte, defaults *schema.GroupVersionKind, into runtime.Object) (runtime.Object, *schema.GroupVersionKind, error) { // Delegate for things other than Unstructured. if _, ok := into.(runtime.Unstructured); !ok && into != nil { return d.delegate.Decode(data, defaults, into) } return unstructured.UnstructuredJSONScheme.Decode(data, defaults, into) } type unstructuredObjectTyper struct { delegate          runtime.ObjectTyper unstructuredTyper runtime.ObjectTyper } func (t unstructuredObjectTyper) ObjectKinds(obj runtime.Object) ([]schema.GroupVersionKind, bool, error) { // Delegate for things other than Unstructured. if _, ok := obj.(runtime.Unstructured); !ok { return t.delegate.ObjectKinds(obj) } return t.unstructuredTyper.ObjectKinds(obj) } func (t unstructuredObjectTyper) Recognizes(gvk schema.GroupVersionKind) bool { return t.delegate.Recognizes(gvk) || t.unstructuredTyper.Recognizes(gvk) </ins> } var crEncoderInstance = crEncoder{}

} } <del> type UnstructuredCreator struct{} </del> <ins> type unstructuredCreator struct{} </ins> <del> func (UnstructuredCreator) New(kind schema.GroupVersionKind) (runtime.Object, error) { </del> <ins> func (c unstructuredCreator) New(kind schema.GroupVersionKind) (runtime.Object, error) { </ins> ret := &unstructured.Unstructured{} ret.SetGroupVersionKind(kind) return ret, nil

type UnstructuredCopier struct{} func (UnstructuredCopier) Copy(obj runtime.Object) (runtime.Object, error) { <ins> if _, ok := obj.(runtime.Unstructured); !ok { // Callers should not use this UnstructuredCopier for things other than Unstructured. // If they do, the copy they get back will become Unstructured, which can lead to // difficult-to-debug errors downstream. To make such errors more obvious, // we explicitly reject anything that isn't Unstructured. return nil, fmt.Errorf("UnstructuredCopier can't copy type %T", obj) } </ins> // serialize and deserialize to ensure a clean copy buf := &bytes.Buffer{} err := unstructured.UnstructuredJSONScheme.Encode(obj, buf)

return result, err } <del> type UnstructuredDefaulter struct{} </del> <ins> type unstructuredDefaulter struct { delegate runtime.ObjectDefaulter } </ins> <del> func (UnstructuredDefaulter) Default(in runtime.Object) {} </del> <ins> func (d unstructuredDefaulter) Default(in runtime.Object) { // Delegate for things other than Unstructured. if _, ok := in.(runtime.Unstructured); !ok { d.delegate.Default(in) } } </ins> type CRDRESTOptionsGetter struct { StorageConfig           storagebackend.Config