Change:
What happened: I use with AWS EKS to authenticate access to clusters. You don't need to worry about how works, the key point is that it prompts the user to enter a 2FA code before running the AWS command that returns a token. When a command is part of a larger shell script, it fails to promp for a 2FA code, and thus fails to authenticate with the cluster. I see error messages like this: For reference, here is what the user looks like in my kube config: What you expected to happen: should forward STDIN to the shell so that it can prompt for a 2FA code. How to reproduce it (as minimally and precisely as possible): Have a kube config user that uses exec, the command should prompt the user for input before returning credentials It would probably be sufficient to have a script that prompts the user and then returns static credentials Run a command within a shell but send STDOUT somewhere else for instance should work Anything else we need to know?: I think I see what's going on. The exec plugin checks whether STDOUT is interactive: but then uses that result to decide whether to forward STDIN: In the case of running as part of a a larger script (where the result is either ignored or e.g. piped to another command), STDIN is still interactive, but STDOUT is not. I didn't find anywhere else where this variable is used, although it's possible I missed something. I also can't think of a case where STDIN would be interactive but you wouldn't want to forward it. Worth noting, based on that code, that STDERR is forwarded regardless. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Amazon EKS 1.18 OS: Mac OS 11.1 Kernel: Install tools: my installation is provided by Docker Desktop if that matters
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/sig auth /cc
/assign
I made a reproducer script[0] for what you are experiencing. When I change that line[1] that mentioned to check if is a terminal, that reproducer script passes. I don't have much historical context on why we were checking instead of . The didn't help either. The API had an field, but that went away in . What about the case where is a pipe (or any other "raw" way of hooking up standard input to ) (i.e., )? would return for that, but would return . Do we want that case to work? Could we check if either_ or are terminals in order to cover this issue's use case while supporting the pipe use case? More generally, should we set (not sure how we would do this?). Aside: we actually are hoping to add some integration or cmd tests that validate that this stdin-hooking-up works properly, so this is a great preview of what the test should do. [0] [1]
Wonder if has any thoughts.
Something occurred to me on now, it kinda seems like a mystery to a running exec plugin whether stdin is hooked up correctly (another person reported something similar). I feel like an exec plugin would like to be able to declare "yes, I use stdin, please forward stdin to me, regardless of what is happening with stdout" in the kubeconfig. Then, this whole stdin forwarding would be a lot less magical and a lot more observable to the exec plugin and the exec plugin author. This makes me wonder why the field was removed from ... do you have any history on why that was? Was there a security worry about this?
in general, I don't see how an exec plugin that consumes stdin can work when combined with uses of kubectl like this: Making plugins that consume stdin declare that they do, and erroring early if something else in the kubectl invocation also indicates it wants to consume stdin seems like a good idea to me.
agreed with the above short term proposal: set to , so this will fix issue, and should provide a proper superset of functionality to exec plugins upon upgrade long term proposal (if this is appetizing, we can move this to the KEP): add and default this field to only forward to the exec plugin when is pro: will be able to force to be forwarded his exec plugin reliably pro: backwards compatible with previous users of pro: plugins can declare when they want to use stdin, so this forwarding is more intentional and observable con: may have to update his kubeconfig with this new field, which may or may not be hard con: we still have this weird logic (can we remove this in ?) con: we may be flapping back to an ExecCredential.Spec.Interactive design that was previously not good? con: stdin can still be stolen from kubectl ( last point above) i started to try to fix that last con, but i couldn't get there...closest i got was this, but i'm not a fan: when knows it will use [0], assert that it is not using an exec plugin that wants to use [0]
Right now I'm making manual changes to my kubeconfig anyway, so having to add another field doesn't sound too bad. The case of piping something to kubectl is an interesting one, and I can see how it's ambiguous in a difficult-to-resolve way.
That's helpful feedback thanks. Another question - do you have a suitable workaround here? The reason why I ask is because I think we would need to do a proper review on the long term proposal above, which would mean it probably wouldn't land until 1.22 (since 1.21 KEP freeze is in 4 days), so if we don't implement the short-term proposal (which seems a little hacky to me?), then this issue wouldn't be fixed until 1.22.
I have an okay workaround for now. The session with MFA is cached so I just need to remember to run some or command directly before running one of these scripts. It's annoying but I can live with it for a while.
Alright here is where I am at with desired behavior (forgetting for a second about backwards compatibility): should only try to hook up stdin to the exec plugin if the exec plugin declares that it wants to use it should return an error if the exec plugin declares that it wants to use stdin and stdin is not a terminal an exec plugin that wants to read from stdin (but did not declare it) tries to read from stdin, we should try to make sure they get an error message that is helpful (e.g., "stdin was not hooked up to this process, please declare it" - not sure we can communicate this error across the process boundary between the client-go process and the exec plugin but i'm sure we can figure something out) Just running through some sharp edges in my head: Long running client-go daemon process that uses an exec plugin that declares it wants to use stdin should fail quickly with an error saying that the process stdin is not a terminal Long running client-go daemon process that uses an exec plugin that does not declare it wants to use stdin but still tries to use stdin should return a helpful error saying that the exec plugin tried to read from stdin and it is not hooked up (again, not sure we can do much here because of the process boundary) Short running kubectl process that uses an exec plugin that does declare it wants to use stdin but takes input from pipe (e.g., ) should fail quickly with an error saying that the process stdin is not a terminal Short running kubectl process that uses an exec plugin that does not declare it wants to use stdin and then tries to use stdin should fail with a helpful error saying that the exec plugin tried to read from stdin and it is not hooked up (again, not sure we can do much here because of the process boundary) Short running kubectl process that uses an exec plugin that does declare it wants to use stdin while kubectl is also using it (e.g., ) should hopefully succeed since the user will input their stuff on stdin to the exec plugin and then input their stuff on stdin to the (this one is nasty) Short running kubectl process that uses an exec plugin that does declare it wants to use stdin while kubectl stdout is being directed somewhere else (e.g., ) should hopefully succeeds since the exec plugin should be writing stuff that it wants the user to see to stderr (e.g., ) (this issue) Short running kubectl process that uses an exec plugin that does declare it wants to use stdin while kubectl stderr is being directed somewhere else (e.g., ) is kinda screwed because the user will never see any prompts (e.g., )... :(
Thanks for thinking through all of that (that reads like a comment that took about three days to write)! Being explicit (in a new client exec config version) seems like a good idea. I think I agree with this. Can we reliably detect stdin is not a terminal? Should client-go be the one to error or should it let the plugin error when reading from stdin fails? I could envision plugins requesting stdin for interactive mode but having a fallback approach if unavailable. If we can reliably detect this, and not send misleading info to the plugin (e.g. hook up a dummy stdin that just returns an EOF error), that maybe seems ok. I think the error is less that stdin is not a terminal and more that two things are trying to consume it.
Appreciate that comment thanks for helping move this issue along. I am really hoping that we can be confident that returns true if and only if stdin is a terminal. I don't know how we would otherwise detect that stdin is a terminal. I'm certainly not a subject matter expert here, just speaking from experience and trusting the package. Good call. Agreed. I think this is captures in proposal below. I hope that will do this for us (see ), but I have only tested on macOS. Mmm, yeah. The "two things are trying to consume it" bit is still bugging me. I haven't figured out how we detect that. I wonder if someone could also run into this with a kubectl plugin that uses if someone built their own command line tool on top of ? pitched me his enum idea which made sense to me. My understanding of his idea was: For beta, add an optional field to the kubeconfig exec config where you can specify whether your exec plugin wants stdin passed to it and the should fail if stdin is not a terminal your exec plugin wants stdin passed to it or your exec plugin wants stdin passed to it if and only if stdin is a terminal, but regardless the exec plugin should always be run For GA, make the field required. The default behavior for this config option in beta would be what we have today (pass stdin to the exec plugin if stdout is a terminal). There was some more conversation in Slack here: I took a pass at implementing the beta functionality here with the hopes that we could submit it as a "bug fix" for 1.21:
OK, one final pass here to see if we can make progress on this in 1.21: posited that we may be able to get away with simply changing the line that originally pointed out. We would still apply a better fix to this in 1.22, but we wanted to hopefully make progress on this in 1.21. Slack convo here:
Thank you to everyone who worked/is working on this. To be honest I didn't expect it to get traction so quickly. Thanks for working on it, but also for taking the time to consider how other people might interact with it and do it the right way.

Discussion:
stdin:       os.Stdin, stderr:      os.Stderr, <del> interactive: term.IsTerminal(int(os.Stdout.Fd())), </del> <ins> interactive: term.IsTerminal(int(os.Stdin.Fd())), </ins> now:         time.Now, environ:     os.Environ,