Code change:
t.Error("unexpected cache keys") } } <ins> func Test_ExecPluginEnvVars(t *testing.T) { testcases := []struct { name            string systemEnvVars   []string execPlugin      *execPlugin expectedEnvVars []string }{ { name:          "positive append system env vars", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin"}, execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, { name:          "positive no env vars provided in plugin", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin"}, execPlugin:    &execPlugin{}, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", }, }, { name: "positive no system env vars but env vars are provided in plugin", execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, { name:            "positive no system or plugin provided env vars", execPlugin:      &execPlugin{}, expectedEnvVars: nil, }, { name:          "positive plugin provided vars takes priority", systemEnvVars: []string{"HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=1111"}, execPlugin: &execPlugin{ envVars: []kubeletconfig.ExecEnvVar{ { Name:  "SUPER_SECRET_STRONG_ACCESS_KEY", Value: "123456789", }, }, }, expectedEnvVars: []string{ "HOME=/home/foo", "PATH=/usr/bin", "SUPER_SECRET_STRONG_ACCESS_KEY=1111", "SUPER_SECRET_STRONG_ACCESS_KEY=123456789", }, }, } for _, testcase := range testcases { t.Run(testcase.name, func(t *testing.T) { testcase.execPlugin.environ = func() []string { return testcase.systemEnvVars } var configVars []string for _, envVar := range testcase.execPlugin.envVars { configVars = append(configVars, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) } merged := mergeEnvVars(testcase.systemEnvVars, configVars) err := validate(testcase.expectedEnvVars, merged) if err != nil { t.Logf("unexpecged error %v", err) } }) } } func validate(expected, actual []string) error { if len(actual) != len(expected) { return errors.New(fmt.Sprintf("actual env var length [%d] and expected env var length [%d] don't match", len(actual), len(expected))) } for i := range actual { if actual[i] != expected[i] { return fmt.Errorf("mismatch in expected env var %s and actual env var %s", actual[i], expected[i]) } } return nil } </ins>

Related discussion:
text: The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.