Code change:
t.Error("unexpected cache keys") } } <ins> func Test_ExecPluginEnvVars(t *testing.T) { testcases := []struct { name            string systemEnvVars   []string execPlugin      *execPlugin expectedEnvVars []string }{ { name:          "positive append system env vars", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin"}, execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, { name:          "positive no env vars provided in plugin", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin"}, execPlugin:    &execPlugin{}, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", }, }, { name: "positive no system env vars but env vars are provided in plugin", execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, { name:            "positive no system or plugin provided env vars", execPlugin:      &execPlugin{}, expectedEnvVars: nil, }, { name:          "positive plugin provided vars takes priority", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=1111"}, execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=1111", "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, } for _, testcase := range testcases { t.Run(testcase.name, func(t *testing.T) { testcase.execPlugin.environ = func() []string { return testcase.systemEnvVars } var configVars []string for _, envVar := range testcase.execPlugin.envVars { configVars = append(configVars, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) } merged := mergeEnvVars(testcase.systemEnvVars, configVars) err := validate(testcase.expectedEnvVars, merged) if err != nil { t.Logf("unexpecged error %v", err) } }) } } func validate(expected, actual []string) error { if len(actual) != len(expected) { return errors.New(fmt.Sprintf("actual env var length [%d] and expected env var length [%d] don't match", len(actual), len(expected))) } for i := range actual { if actual[i] != expected[i] { return fmt.Errorf("mismatch in expected env var %s and actual env var %s", actual[i], expected[i]) } } return nil } </ins>

Related discussion:
text: Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?