Change:
Using this script: I'm seeing some random errors. 1 - running the above script it fails for me on the near the bottom when it tries to create . Notice that I had already successfully created which of the same type: Sometimes if I wait a while (like over a minute) then the two commands at the bottom work. 2 - At the top, if I comment out the commands and uncomment the commands, it fails with: Which is interesting, because when I give it a totally unknown type I get this: which leads me to think that it does know about the Org1Type but fails late in the processing for some reason. I said these errors we random, because there have been times when I would run this script and then go back later and try some of the failing commands manually and they would work - almost like there's a timing/async issue going on - but since I couldn't reliably reproduce those all I can do is just mention it but can't provide a solid testcase to show it, yet. Additionally, I've seen cases where the (for one of the instances) would fail and then I would try again and it would complain that the resource I'm trying to create already exists. I'm wondering if this is due to the original create actually working but then failing on some kind of verification step afterwards and that's the error being shown - but that's just a guess. I'm running these tests with PR but I saw seeing similar results even from master. ping
Any chance addresses this?
I'm pretty sure I tried that PR w/o any luck but let me try again just to be
nope, still fails for me using . On a related question... is it normal for thirdPartyResource types to not be visible right away? I find that I have to add a in there to wait for these new types to be known. I understand that some things in kube need/should-be async, but I'm not sure why the creation of these would be.
Hm, I'd have to look at the code, I thought it should be synchronous, but maybe not? might know.
It is not synchronous. It is picked up by the thirdparty controller. It can take a little while.
is something that can be changed or does it need to be this way?
That would take a lot more effort I'd say. Cannot comment about the feasibility, as I have not looked into that option yet.
Also note that if you're in a HA system, you may need to wait until it shows up on all apiserver replicas before using it... , Uday Ruddarraju wrote:
Is there a way we can tell if that has happened? It seems like we need some better way to check than "hit it until it works, and even then, maybe it does not".
An option is that we can wait until that really happen.
Yes I would prefer for certain operations to wait until they're really done before they return but as asked, how can we know for sure in this case?
I see some options: wait to return until it's ready to be called. return some value on where to check for it to be ready. resource is stubbed out immediately, and returns some 'not yet ready' status until it is. knows enough to 'keep trying' until some timeout. I'm not sure what the intention is going forward if in fact core k8s APIs are going to be switched to using this mechanism. It can't be left in such an inconsistent state without specification. This behavior is not in the design.
IMO, kubectl can try to check some condition until some timeout, but we still need to make sure what's the condition, I would hava to take a look at the thirdparty resource code and see what we can do.
I think doing it on the kubectl side would be a mistake because then the REST API clients would need to replicate that logic. If there are certain operations (like, IMO, creating a new 3rd party resource) that should be blocked then I'd prefer if the POST itself blocked until it was done.
The logical here is that when you create a new ThirdPartyResource resource, it will return just after the resource is stored in etcd, but the process to register a new RESTful resource path and add a new api group is asynchronous, this is what thirdparty_controller will do. Basically put this wait in kubectl is wrong, this makes kubectl create complicate. As for real use case, users can use discovery client to verify that the api group is registered.
Third party resource installation is not meant to be flipped on and off often. The only use case for that is testing your third party bring up. And there it's sufficient to just keep listing the third party data path until it returns a success value. For extra certainty, look up the endpoints of the kubernetes service and make sure it appears on every endpoint. The add is an async operation. It returns success because the 3rd party resource spec has been saved. I definitely think we should document the need for this wait better but I don't think we should actually make the control plane do the wait for you.
That's just what I mean, we should document this
fwiw, the async nature is intentional. the system is purposely de-coupled, there are lots of other examples of this in the system (e.g. creating a replication controller doesn't create pods immediately) It's much more robust this way (of course it does put a little more burden on the client side) I could see adding a utility function somewhere to wait, but honestly it's basically
yep, I get that and for lots of things the async nature makes sense. But I'm not sure it makes sense for all. For example, I think people would understand that it takes a while to spin up containers, but I'm not so sure the delay for creating a new resource type (which is mainly updating etcd) is so easy to understand. I think, at an abstract level, a lot of this goes to the UX. Sure, every client could do that 404-loop you wrote, but is that really necessary in all cases and is that the best UX we can offer? From a UX perspective I think there are going to be some operations that should appear to be synchronous even if under the covers its async and the server does the 404-loop for you (or at least by default and have a flag to disable it). And let's not forget there's nothing stopping a client from doing the async work on their side by running their cmd in the background, on a new thread/process, and then checking for that thread to complete - if they really didn't want to wait around. So, net, I think we need to look at this from a user's perspective and not a Kube-developer's, who understands what's going on under the covers. From a user's perspective, for operation "X" would I normally expect that to take some time? If so, then an async UX is ok. But if I would expect "X" to be immediate (meaning once I get back the 200 I should be able to take the next action on that resource because I should have gotten back a non-200 immediately if there was an error) then exposing it as an async op might not be the best UX. IMO, creating a TPR falls into that category.
The fact that there can be multiple apiservers means it really is an async operation: you need to wait for each apiserver to get the watch notification and install the handlers. Operations with that sort of dependency are never synchronous in our api. Is it problematic to just do the wait loop, or just wait 10 seconds or something? How often are you adding new 3rd party resources? , Doug Davis wrote:
(I'm not arguing that we shouldn't make things easy for users, rather that we are working on other things that are even worse for users and this is less important since it should be used infrequently.) , Daniel Smith wrote:
Just as background on my current experimentation with TPRs: I'm using a single hyperkube env, which I think has just a single API server. In this work, which is actually part of an education class, we're walking thru a script that registers some TPRs and then creates instances of them to show students about the concept and how to use them. I'm finding at after each I have to add some kind of or , that's a pretty poor UX. I can certainly understand in a multi-api-server env there may be more complex issues involved, but if we can at least improve it for the simple case I think that would help a lot. I would also note that even a !404-loop isn't sufficient in the HA case since you may be getting luck w.r.t. which apiServer you're hitting, right? And yes, I know there are a ton of bigger fish to fry and I do appreciate the discussion. :-)
It's unfortunately not really easier to do it for the single-master case, since we have to support the multi-master case. If we add an exception for the single master case that adds complexity and actually makes the problem a bit worse for the multi-master case. The "Kubernetes-y" thing to do here would be to add a status component to the 3rd party resource object and have each apiserver report when they've applied the change, then you'd wait for that instead of your current 404-loop. Having the POST method itself do the waiting is going to be a really tough sell. Watch is intended to handle all corner cases involved when clients are waiting on information from the server, POST is not equipped to do that. (e.g., what happens if the apiserver you POSTed to crashes while waiting? You wouldn't know if you should re-post or not and you wouldn't have any way to figure out the status at that point.) , Doug Davis wrote:
I wouldn't argue for a single-master exception, nor would I argue for a POST that waits. I am unfamiliar with how everything works, so I am having a little trouble parsing some of these responses. I see mentioned a and . What are those? Are they applicable here? Is there a way to watch for this change to be made that is built in, or is the way to go for now?
Why was this closed? is a doc issue, I don't think this is a doc problem.
I am not so concerned about the UX issue if the delay really is isolated to system start-up (as if there is such a thing). What about this scenario: operator registers a TPR, then waits for all the apiservers running at the time to fully react, and thereafter nothing is prepared for the TPR to be absent. Some time later another apiserver is , or an existing one crashes and is re-started. Will there be an interim during which the new(ly started) apiserver is not ready for clients to use the TPR?
would know better, but I'd expect the readiness check to fail until the initial sync of registered apiservice objects completed, preventing the new API server from making it into the LB pool
will report unhealthy until the apiservices are synced, but TPR was based on an unusual polling technique that can take about 10 seconds. I'm not certain when it will be ready. CRD is based on the state of an informer's list/watch, so its much more current.
This consistently works well for CRDs. modified script to use CRDs:
Closing since this works for CRDs. /close

Discussion:
Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`: