jiebi/CodeConvo huggingface dataset:
Code change:
"sync/atomic" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> discovery "k8s.io/api/discovery/v1alpha1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.
Code change:
"sync/atomic" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> discovery "k8s.io/api/discovery/v1alpha1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: ): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWSPROFILE which resides in the default () directory. The profile references a which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs: However, when I type the following into the command line on the kubelet: I get the following response: It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary. Below is Config file: I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary. I am using KinD to test this feature. Here is my config: No Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 20.10 Kernel (e.g. ): 5.10 Install tools: KinD Network plugin and version (if this is a network-related bug): Others: cc
/sig Node
/assign
from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!!
Thanks Yes, I saw that. doesn't work. I wrote the following code to test this: If I uncomment , which imports the environment variables from the OS, it works.
Hello need some clarification. Is the intent of the above code to convey that isn't passing env vars correctly to the spawned process? Below code works as expected for me Also, could you tell if is a proprietary binary or a open source code something similar to because I want to use it in troubleshooting this issue.
Yes it is not passing the Envs to the spawned process.
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
why does it work when I run it from the command line? Why does it work when I import envs from the OS?
you might have saved env on command line, same goes for OS.
That doesn't make sense to me. If I run it returns a properly formatted response. Same goes for the Go code I wrote. Does calling another binary, like credentialprocess, confuse the kubelet?
I am not familiar with aws credential process. Kubelet talks to with exec based plugin mechanism, is ecr-credential-provider calling the credential process you are talking about?
I have already validated that env vars are being passed to and there is no issue over there. If that wouldn't be the case then AWS credentials generation process via wouldn't work for you. I had debugged the code of kubelet myself and validated it + I wrote some logs in to print out the env vars and it was logging the passed vars. IMHO, now what remains to be seen here is why it fails with Cognito. I am going to create a test bed tomorrow replicating the exact setup which you have in the mean time if it's possible for you could you create a Linux VM and replicate your setup inside it? My intention is to see whether the Cognito based credential process goes through or not in a clean slate environment if it does then something on your local setup is interfering with passed env vars Thoughts?
Here's the execution flow ---- Does the above paints a fairly accurate picture? If not then feel free to correct :)
I included my KinD configuration in the beginning of this thread. Can you use that The binary has to be built from source, (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at The flow is as describes above. The binary returns a json blob to the calling application that looks like this: During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using or by setting to . However, when I try to pass the AWS profile in using exec.Env("AWS_PROFILE=default") I get and no further information. I have a feeling that doesn't properly chain or pass information back to the calling application.
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS file look like? Is it something like this You can use placeholder values for sensitive information but correct the binary path if it's not
yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation.
Yes, I am planning to setup Cognito Identity pool in my AWS Account.
So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via I am calling a binary written by me which logs env vars to a specific path and prints out credentials in the format as prescribed by AWS. Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my vars section of my deployment spec like below and it worked The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of from start up until you create a Pod?
thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results.
Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using .
^^ is my credential provider config
Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds.
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: If it doesn't work out then I'll reach out to you for creds
After several rounds of troubleshooting I was able to get working in my KinD environment. My credential loading setup is <-<-and below is my configuration of credential process Initially, I got the same error as yours, then I couple of custom log lines to print env vars in in directory and I could verify that the env vars specified in the credentials spec were indeed being passed. What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the container that file and were indeed present. I altered my env vars section of the as below and it worked for me Apparently, what I noticed is is being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster. Can you please add and other env vars necessary for the correct execution of AWS SDK and see if it works for you? If this doesn't work, then I'll send you the modified version of the which puts some additional logs to troubleshoot this issue further.
That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future.
What I have observed is, is only passed env vars which are in credential spec and the spec clearly says that the vars in spec are appended to existing env vars and this looks like an issue which needs deeper investigation. Could you please re-open this issue?
sure thing.
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
To me this looks like an issue with the kubelet and I have observed this behaviour with as well as my local cluster. What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this.
Code change:
imageParts := strings.Split(image, "/") return imageParts[0] } <ins> // mergedEnvVars overlays system defined env vars with credential provider env vars, // it gives priority to the credential provider vars allowing user to override system // env vars func mergeEnvVars(sysEnvVars, credProviderVars []string) []string { mergedEnvVars := sysEnvVars for _, credProviderVar := range credProviderVars { mergedEnvVars = append(mergedEnvVars, credProviderVar) } return mergedEnvVars } </ins>

Related discussion:
text: Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it /triage accepted
Thanks!!!
/area kubelet
: I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me
I've been fighting this for days - did you ever figure it out?
Code change:
defer f.Unlock() f.CalledFunctions = append(f.CalledFunctions, "ListImages") <del> return f.ImageList, f.Err </del> <ins> return snapshot(f.ImageList), f.Err } func snapshot(imageList []kubecontainer.Image) []kubecontainer.Image { result := make([]kubecontainer.Image, len(imageList)) copy(result, imageList) return result </ins> } func (f *FakeRuntime) RemoveImage(_ context.Context, image kubecontainer.ImageSpec) error {

Related discussion:
text: when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change
Code change:
"//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/json:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/client-go/util/workqueue:go_default_library", "//vendor/k8s.io/klog:go_default_library", "//vendor/sigs.k8s.io/yaml:go_default_library",

Related discussion:
text: There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help