Repository: external-secrets/external-secrets (Go)

Issue:
ExternalSecret that refer ECRAuthorizationToken not work in 0.18.0
**Describe the bug**
After we upgrade the external-secrets version to 0.18.0, the externalsecret that refer the ECRAuthorizationToken report the error, but it works with version 0.17.0(refer the manifest below, it works). 
looks like it will not use the IRSA permission to perform the `ECR: GetAuthorizationToken`. The logs find in controller:

```
{"level":"error","ts":1750228541.0351596,"msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"repository-ecr","namespace":"argocd"},"namespace":"argocd","name":"repository-ecr","reconcileID":"xxxxxxxxx","error":"error processing spec.dataFrom[0].sourceRef.generatorRef, err: error using generator: unable to get authorization token: operation error ECR: GetAuthorizationToken, https response error StatusCode: 400, RequestID: xxxxxxxxx, api error MissingAuthenticationTokenException: Missing Authentication Token","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}
```

**To Reproduce**
Steps to reproduce the behavior:
1. The ECRAuthorizationToken manifest

```yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-auth-token
  namespace: argocd
spec:
  region: ap-southeast-2
```

2. The ExternalSecret manifest

```yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: repository-ecr
  namespace: argocd
spec:
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: ecr-auth-token
  refreshInterval: 30m
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    template:
      data:
        enableOCI: "true"
        password: "{{ .password }}"
        type: helm
        url: 11111111111.dkr.ecr.ap-southeast-2.amazonaws.com
        username: "{{ .username }}"
      engineVersion: v2
      mergePolicy: Replace
      metadata:
        labels:
          argocd.argoproj.io/secret-type: repo-creds
      type: Opaque
```

The AWS role permission:

<img width="873" alt="Image" src="https://github.com/user-attachments/assets/60162520-bbe3-4a6f-8172-fa8b3a086620" />

Kubernetes version: 1.31(EKS)
ESO version: 0.18.0

**Expected behavior**
The `ExternalSecret` should works fine with ECRAuthorizationToken. I know there are some refactor of AWS, but from the [docs](https://external-secrets.io/latest/api/generator/ecr/), not mention the manifest need some change(diff the old version with the latest).



Discussion:
we're in the same boat, also experiencing this issue..
can you all try to add a `AWS_REGION` environment variable to your external-secrets pod to see if it works as a workaround? I think this might be related with #4934 
> can you all try to add a `AWS_REGION` environment variable to your external-secrets pod to see if it works as a workaround? I think this might be related with [#4934](https://github.com/external-secrets/external-secrets/issues/4934)

I suppose we should have mentioned, we already use IRSA for the controller. As in, not fine grained IRSA associated with each SecretStore.

So, due to how IRSA works, the controller Pods already have `AWS_REGION` env var. The env section for controller Pods look like:

```yaml
    env:
    - name: AWS_STS_REGIONAL_ENDPOINTS
      value: regional
    - name: AWS_DEFAULT_REGION
      value: ap-southeast-2
    - name: AWS_REGION
      value: ap-southeast-2
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::{accountId}:role/{role-name}
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
```
Ugh, most likely something with AWS SDK v2. @Ilhan-Personal any ideas? 🙇 
Same issue here, for context we use Pod Identity, and this is the error I get on the daemonset:

```
eks-pod-identity-agent-tvscr eks-pod-identity-agent {"association-id":"a-1tlv7jypxfjslotbc","client-addr":"10.144.2.172:36220","cluster-name":"test1","from":"renewal-thread","level":"info","msg":"Removing credentials from cache, got non recoverable error: error getting credentials to cache: unable to fetch credentials from EKS Auth: operation error EKS Auth: AssumeRoleForPodIdentity, https response error StatusCode: 400, RequestID: e8233c5e-5d65-4355-bb64-39d6158b4eef, ExpiredTokenException: The token included in the request is expired: current date/time 2025-06-18T21:20:55.347735677Z must be before the expiration date/time 2025-06-18T20:45:00Z.","time":"2025-06-18T21:20:55Z"}
```

Looking
Thanks! :) Maybe it's scope related? Maybe try setting public scope? 🤔 
Heyy @Skarlso why do you think it would be scope related?

also QQ, I noticed that the function to return webidentityroleprovider accepts an argument for region but it looks like I probably missed to include the region while setting up the config, But yes I'm not 100% sure I thought the credential chain would handle this since we're specifying the env var anyway 

https://github.com/external-secrets/external-secrets/pull/4940/files#diff-7716174091c7f7fad44cb0df854f13815330251a6d5bb610fc6966c74dcc9789R301

EDIT: 

The above is probably not related to the current issue but it could be a fix for a different issue 
https://github.com/external-secrets/external-secrets/issues/4934
Got the same issue:

```yaml
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-gen
spec:
  region: sa-east-1
  auth:
    secretRef:
      accessKeyIDSecretRef:
        name: "ecr-creds"
        key: "AWS_ACCESS_KEY_ID"
      secretAccessKeySecretRef:
        name: "ecr-creds"
        key: "AWS_SECRET_ACCESS_KEY"
```

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ecr-creds
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: ecr-creds
    creationPolicy: Owner
  data:
  - secretKey: AWS_ACCESS_KEY_ID
    remoteRef:
      key: ecr-auth
      property: AWS_ACCESS_KEY_ID
  - secretKey: AWS_SECRET_ACCESS_KEY
    remoteRef:
      key: ecr-auth
      property: AWS_SECRET_ACCESS_KEY
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: "ecr-auth"
spec:
  refreshInterval: "1h"
  target:
    name: ecr-auth
    template:
      type: kubernetes.io/dockerconfigjson
      metadata:
        annotations:
          expiresAt: "{{ .expires_at }}"
      data:
        .dockerconfigjson: |
          {
            "auths": {
             {{ "{{ .proxy_endpoint | replace \"https://\" \"\" }}" }}: {
                "username": "{{ .username }}",
                "password": "{{ .password }}",
                "auth": "{{ printf "%s:%s" .username .password | b64enc }}"
              }
            }
          }
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: ECRAuthorizationToken
          name: "ecr-gen"
```

This is the error I get
```
Secret "ecr-auth" is invalid: data[.dockerconfigjson]: Invalid value: "<secret contents redacted>": invalid character '9' looking for beginning of object key string
```


This is the live manifest I see in argocd

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"nw-backend"},"name":"ecr-auth","namespace":"nw"},"spec":{"dataFrom":[{"sourceRef":{"generatorRef":{"apiVersion":"generators.external-secrets.io/v1alpha1","kind":"ECRAuthorizationToken","name":"ecr-gen"}}}],"refreshInterval":"1h","target":{"name":"ecr-auth","template":{"data":{".dockerconfigjson":"{\n 
      \"auths\": {\n   {{ .proxy_endpoint | replace \"https://\" \"\" }}:
      {\n      \"username\": \"\",\n      \"password\": \"\",\n      \"auth\":
      \"JSFzKDxuaWw+KTolIXMoPG5pbD4p\"\n    }\n 
      }\n}\n"},"metadata":{"annotations":{"expiresAt":""}},"type":"kubernetes.io/dockerconfigjson"}}}}
  creationTimestamp: '2025-06-21T21:25:48Z'
  generation: 4
  labels:
    app.kubernetes.io/instance: nw-backend
  name: ecr-auth
  namespace: nw
  resourceVersion: '4481308'
  uid: aeaec56c-1e28-4e17-b891-13de213fedd0
spec:
  dataFrom:
    - sourceRef:
        generatorRef:
          apiVersion: generators.external-secrets.io/v1alpha1
          kind: ECRAuthorizationToken
          name: ecr-gen
  refreshInterval: 1h
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: ecr-auth
    template:
      data:
        .dockerconfigjson: |
          {
            "auths": {
             {{ .proxy_endpoint | replace "https://" "" }}: {
                "username": "",
                "password": "",
                "auth": "JSFzKDxuaWw+KTolIXMoPG5pbD4p"
              }
            }
          }
      engineVersion: v2
      mergePolicy: Replace
      metadata:
        annotations:
          expiresAt: ''
      type: kubernetes.io/dockerconfigjson
status:
  binding:
    name: ''
  conditions:
    - lastTransitionTime: '2025-06-21T21:25:48Z'
      message: could not update secret
      reason: SecretSyncedError
      status: 'False'
      type: Ready
  refreshTime: null
```

And if we decode `JSFzKDxuaWw+KTolIXMoPG5pbD4p` we get `!s(<nil>):%!s(<nil>)` which is the same thing I see in logs

to me it just looks like it didn't work at all and documentation has no examples (and one example with bad quotes that took me forever to debug)
> Got the same issue:
> 
> apiVersion: generators.external-secrets.io/v1alpha1
> kind: ECRAuthorizationToken
> metadata:
>   name: ecr-gen
> spec:
>   region: sa-east-1
>   auth:
>     secretRef:
>       accessKeyIDSecretRef:
>         name: "ecr-creds"
>         key: "AWS_ACCESS_KEY_ID"
>       secretAccessKeySecretRef:
>         name: "ecr-creds"
>         key: "AWS_SECRET_ACCESS_KEY"
>  apiVersion: external-secrets.io/v1beta1
> kind: ExternalSecret
> metadata:
>   name: ecr-creds
> spec:
>   refreshInterval: 1m
>   secretStoreRef:
>     name: aws-secrets-manager
>     kind: ClusterSecretStore
>   target:
>     name: ecr-creds
>     creationPolicy: Owner
>   data:
>   - secretKey: AWS_ACCESS_KEY_ID
>     remoteRef:
>       key: ecr-auth
>       property: AWS_ACCESS_KEY_ID
>   - secretKey: AWS_SECRET_ACCESS_KEY
>     remoteRef:
>       key: ecr-auth
>       property: AWS_SECRET_ACCESS_KEY
> ---
> apiVersion: external-secrets.io/v1beta1
> kind: ExternalSecret
> metadata:
>   name: "ecr-auth"
> spec:
>   refreshInterval: "1h"
>   target:
>     name: ecr-auth
>     template:
>       type: kubernetes.io/dockerconfigjson
>       metadata:
>         annotations:
>           expiresAt: "{{ .expires_at }}"
>       data:
>         .dockerconfigjson: |
>           {
>             "auths": {
>              {{ "{{ .proxy_endpoint | replace \"https://\" \"\" }}" }}: {
>                 "username": "{{ .username }}",
>                 "password": "{{ .password }}",
>                 "auth": "{{ printf "%s:%s" .username .password | b64enc }}"
>               }
>             }
>           }
>   dataFrom:
>     - sourceRef:
>         generatorRef:
>           apiVersion: generators.external-secrets.io/v1alpha1
>           kind: ECRAuthorizationToken
>           name: "ecr-gen"
> This is the error I get
> 
> ```
> Secret "ecr-auth" is invalid: data[.dockerconfigjson]: Invalid value: "<secret contents redacted>": invalid character '9' looking for beginning of object key string
> ```
> 
> This is the live manifest I see in argocd
> 
> apiVersion: external-secrets.io/v1beta1
> kind: ExternalSecret
> metadata:
>   annotations:
>     kubectl.kubernetes.io/last-applied-configuration: >
>       {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"nw-backend"},"name":"ecr-auth","namespace":"nw"},"spec":{"dataFrom":[{"sourceRef":{"generatorRef":{"apiVersion":"generators.external-secrets.io/v1alpha1","kind":"ECRAuthorizationToken","name":"ecr-gen"}}}],"refreshInterval":"1h","target":{"name":"ecr-auth","template":{"data":{".dockerconfigjson":"{\n 
>       \"auths\": {\n   {{ .proxy_endpoint | replace \"https://\" \"\" }}:
>       {\n      \"username\": \"\",\n      \"password\": \"\",\n      \"auth\":
>       \"JSFzKDxuaWw+KTolIXMoPG5pbD4p\"\n    }\n 
>       }\n}\n"},"metadata":{"annotations":{"expiresAt":""}},"type":"kubernetes.io/dockerconfigjson"}}}}
>   creationTimestamp: '2025-06-21T21:25:48Z'
>   generation: 4
>   labels:
>     app.kubernetes.io/instance: nw-backend
>   name: ecr-auth
>   namespace: nw
>   resourceVersion: '4481308'
>   uid: aeaec56c-1e28-4e17-b891-13de213fedd0
> spec:
>   dataFrom:
>     - sourceRef:
>         generatorRef:
>           apiVersion: generators.external-secrets.io/v1alpha1
>           kind: ECRAuthorizationToken
>           name: ecr-gen
>   refreshInterval: 1h
>   target:
>     creationPolicy: Owner
>     deletionPolicy: Retain
>     name: ecr-auth
>     template:
>       data:
>         .dockerconfigjson: |
>           {
>             "auths": {
>              {{ .proxy_endpoint | replace "https://" "" }}: {
>                 "username": "",
>                 "password": "",
>                 "auth": "JSFzKDxuaWw+KTolIXMoPG5pbD4p"
>               }
>             }
>           }
>       engineVersion: v2
>       mergePolicy: Replace
>       metadata:
>         annotations:
>           expiresAt: ''
>       type: kubernetes.io/dockerconfigjson
> status:
>   binding:
>     name: ''
>   conditions:
>     - lastTransitionTime: '2025-06-21T21:25:48Z'
>       message: could not update secret
>       reason: SecretSyncedError
>       status: 'False'
>       type: Ready
>   refreshTime: null
> And if we decode `JSFzKDxuaWw+KTolIXMoPG5pbD4p` we get `!s(<nil>):%!s(<nil>)` which is the same thing I see in logs
> 
> to me it just looks like it didn't work at all and documentation has no examples (and one example with bad quotes that took me forever to debug)


Your problem seems to be something with how you’re injecting the manifests than this bug. The live manifests should contain the templates; the fact they don’t have them means you’re probably using a helm renderer 🙂, which is unrelated
@maodahua can you verify if this image  ghcr.io/external-secrets/external-secrets:main@sha256:50a6d9b763b34e4afe9c4a4b7e87508c8bef29ffa8b5140de9d449dd726e50cb fixes this issue?

Image ref: https://github.com/external-secrets/external-secrets/actions/runs/15808006710/job/44555482486
> [@maodahua](https://github.com/maodahua) can you verify if this image ghcr.io/external-secrets/external-secrets:main@sha256:50a6d9b763b34e4afe9c4a4b7e87508c8bef29ffa8b5140de9d449dd726e50cb fixes this issue?
> 
> Image ref: https://github.com/external-secrets/external-secrets/actions/runs/15808006710/job/44555482486

We experienced a similar issue (with a generic secret store, no ECRAuthorizationToken):

```
 "msg":"unable to validate store","secretstore":{"name":"foo","namespace":"foo"},"error":"could not validate provider: failed to refresh cached credentials, failed to retrieve credentials, operation error STS: AssumeRoleWithWebIdentity, failed to resolve service endpoint, endpoint rule error, Invalid Configuration: Missing Region"
```

The suggested image above seems to have fixed the problem.
> [@maodahua](https://github.com/maodahua) can you verify if this image ghcr.io/external-secrets/external-secrets:main@sha256:50a6d9b763b34e4afe9c4a4b7e87508c8bef29ffa8b5140de9d449dd726e50cb fixes this issue?
> 
> Image ref: https://github.com/external-secrets/external-secrets/actions/runs/15808006710/job/44555482486

Hi @gusfcarvalho I am on leave till this Wed, will test it when I am back 🚀 
Hi @gusfcarvalho, faced a similar issue with an AWS secret store and using the above image `ghcr.io/external-secrets/external-secrets:main` from #4940 resolved the `Invalid Configuration: Missing Region` error just to also confirm the changes
`SecretStore`s were already fixed; I'd particularly like to know if this also fixes this `ECRAuthorizationToken` 
I installed 0.18.0 yesterday and I was getting the Missing Region error still which brought me here
I'll create a patch release today.
> [@maodahua](https://github.com/maodahua) can you verify if this image ghcr.io/external-secrets/external-secrets:main@sha256:50a6d9b763b34e4afe9c4a4b7e87508c8bef29ffa8b5140de9d449dd726e50cb fixes this issue?
> 
> Image ref: https://github.com/external-secrets/external-secrets/actions/runs/15808006710/job/44555482486

Hi @gusfcarvalho , sorry for the late reply.

We deploy the controller, cert controller and webhook components. And I take a look of the hem chart, they use the same image by default, so I replace the three images to `ghcr.io/external-secrets/external-secrets:main@sha256:50a6d9b763b34e4afe9c4a4b7e87508c8bef29ffa8b5140de9d449dd726e50cb`

And create the `ECRAuthorizationToken` and `ExternalSecret` use the yaml in describe.

And I still got issue.

```
{"level":"error","ts":1750929255.9652507,"msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"repository-ecr","namespace":"default"},"namespace":"default","name":"repository-ecr","reconcileID":"xxxxxxxxx","error":"error processing spec.dataFrom[0].sourceRef.generatorRef, err: error using generator: unable to get authorization token: operation error ECR: GetAuthorizationToken, https response error StatusCode: 400, RequestID: xxxxxxxxx, api error MissingAuthenticationTokenException: Missing Authentication Token","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202"}
```

And after I change the image back to `oci.external-secrets.io/external-secrets/external-secrets:v0.17.0`, the `ExternalSecret` works good.


Confirming this is still a problem with 0.18.1 and 0.18.2
@josemaia Seems like for others the fix worked, or did it not? Asking others, I'm not a 100% sure what still the problem is.
Hi @Skarlso, I believe the region fix implemented in .1 might have resolved a **subset** of the problem but not the whole thing.

I'm trying to use this scenario with Web Identity on the External Secrets Controller (Azure injects a web identity token, then the AWS SDK is used to assume a role via OIDC). I got it to work in a random pod with AWS CLI, and in ecr-credential-helper, both via environment variables AWS_WEB_IDENTITY_TOKEN_FILE / AWS_ROLE_ARN, or AWS_CREDENTIAL_PROFILES_FILE and a config/credentials file setup with the role ARN and token file path.

i'm running external-secrets 0.10 in production, and this strategy did not work, but with a ValidationError.

For 0.10, I had these errors:
```bash
{"level":"error","ts":1751470168.5552232,"logger":"controllers.ExternalSecret","msg":"could not get secret data from provider","ExternalSecret":{"name":"ecr-secret","namespace":"logging"},"error":"could not generate [0]: unable to get authorization token: ValidationError: josemaia-ecr-azure-test is invalid\n\tstatus code: 400, request id: 2773e2f4-4d2f-4218-9384-8620e5ca25f0","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).markAsFailed\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:357\ngithub.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222"}
```

Trying to solve this I upgraded to .18, and instead got the error mentioned by @maodahua. Dropping to .17, that error disappears and the "ValidationError" happens again, which I will continue to dig into and open a separate issue for if I continue having trouble. I believe .18 introduced something to cause this MissingAuthenticationTokenException.

EDIT: I can confirm the ValidationError has absolutely nothing to do with the aforementioned issue, and was able to resolve it.
@Skarlso  Confirmed this is a separate issue than the SecretStore. People here thatn mentioned it now started to work weren't actually using Generators, but instead SecretStores...
Soooo, is this issue then still relevant, or do we need to open a new one with more context?
still relevant i'd say, the Generators can't log in to ECR (at least with web identity) after the SDK upgrade
Looking at pkg/provider/aws/auth/auth.go, I suspect this may have to do with defaults not being properly initialized

In sdk v1 version, there was a line in method NewGeneratorSession() like
```go
sess, err := getAWSSession(config, false, "", "", "", "")
```

In the v2 version, the method New() eventually invokes config.LoadDefaultConfig(), if there is neither a service account nor a static credentials set, but NewGeneratorSession() which is the one used by the generators, doesn't seem to ever do that from what I understand.
Yep, that does it.

[This code](https://github.com/external-secrets/external-secrets/commit/9637ad347ae8268b35bc99887d05ff659a7a9dab) is written purely to validate assumptions, I think maybe a refactor to share more code with New() might be the best way to fix the problem, but by adding this my generator started working.

Maybe @Ilhan-Personal can give some guidance on what would be the best way to integrate such a fix?
Heyy sorry folks about the late reply, I was on a vacation but yes this is indeed a bug introduced by me when I did the refactor.

`Looking at pkg/provider/aws/auth/auth.go, I suspect this may have to do with defaults not being properly initialized`

@josemaia The fix from you sounds right, I've left one comment...Thank you !


opened a draft PR after the discussion on my commit, but I'm gonna need some help to get this one past the finish line (tests, linting, etc.). it works fine in-cluster in the scenarios I tested (generator with role specified or without).
Also running into this, is there any workaround besides downgrading?
> Also running into this, is there any workaround besides downgrading?

I believe the only auth method compromised is Pod Identity.

So, a few options:
* you can use other auth methods like AWS programmatic keys (That is, if I am not mistaken. I no longer have an AWS account to test these types of bugs);
* You can fork the draft PR above and build the code from there, that should give you a hotfix-like setup.
* You can also have a vendor provide and maintain this hotfix for you (i.e. the "throw money at the problem"). 

I guess It depends on how urgent this issue is for you.
I am really glad to see this issue got resolved. Thank you for everyone who worked on this! When can we expect a new release with this fix?
Unsure yet. I might create a patch release this week. But since we are gearing towards v1 we might not do a release for a while. I'll see what the other maintainers will say about it.
JFYI - I've just bumped in to this on 0.16.2 of the helm chart. Is that due to the underlying image version being used?
> JFYI - I've just bumped in to this on 0.16.2 of the helm chart. Is that due to the underlying image version being used?

If you are using helm chart `v0.16.2` but are setting the external-secrets image to be `0.18...`, yes.

By the way, a fair warning heads up - that's unsupported. We do not encourage users to decouple helm chart versions from app versions. In our own opinions, these are coupled and should be considered as one Deployment.

I.e. upgrading the binary image without upgrading the helm chart version will likely break your setup catastrophically 😄  specially if we add changes like new API spec.
> Unsure yet. I might create a patch release this week. But since we are gearing towards v1 we might not do a release for a while. I'll see what the other maintainers will say about it.

Would love if there is patch release, for me burning issue with ECR and ESO
We’ve just realized a patch release will not really be an option; as our SecretStore CRD manifest has grown too big to be applied client-side.

This means a lot of tooling will break OOTB, like ArgoCD
> > JFYI - I've just bumped in to this on 0.16.2 of the helm chart. Is that due to the underlying image version being used?
> 
> If you are using helm chart `v0.16.2` but are setting the external-secrets image to be `0.18...`, yes.
> 
> By the way, a fair warning heads up - that's unsupported. We do not encourage users to decouple helm chart versions from app versions. In our own opinions, these are coupled and should be considered as one Deployment.
> 
> I.e. upgrading the binary image without upgrading the helm chart version will likely break your setup catastrophically 😄 specially if we add changes like new API spec.

I don’t think that’s what I’m doing, I think I was just using the chart with no extra version specified. I’ll double check on Monday.



Patch:
diff --git a/pkg/controllers/secretstore/common.go b/pkg/controllers/secretstore/common.go
index 37d8d744d6a..0943d7ba947 100644
--- a/pkg/controllers/secretstore/common.go
+++ b/pkg/controllers/secretstore/common.go
@@ -124,9 +124,15 @@ func validateStore(ctx context.Context, namespace, controllerClass string, store
 	}
 	validationResult, err := cl.Validate()
 	if err != nil && validationResult != esapi.ValidationResultUnknown {
-		cond := NewSecretStoreCondition(esapi.SecretStoreReady, v1.ConditionFalse, esapi.ReasonValidationFailed, fmt.Sprintf(errUnableValidateStore, err))
+		// Use ReasonInvalidProviderConfig for validation errors that indicate
+		// invalid configuration (like empty server URL)
+		reason := esapi.ReasonValidationFailed
+		if validationResult == esapi.ValidationResultError {
+			reason = esapi.ReasonInvalidProviderConfig
+		}
+		cond := NewSecretStoreCondition(esapi.SecretStoreReady, v1.ConditionFalse, reason, fmt.Sprintf(errUnableValidateStore, err))
 		SetExternalSecretCondition(store, *cond, gaugeVecGetter)
-		recorder.Event(store, v1.EventTypeWarning, esapi.ReasonValidationFailed, err.Error())
+		recorder.Event(store, v1.EventTypeWarning, reason, err.Error())
 		return fmt.Errorf(errValidationFailed, err)
 	}
 
diff --git a/pkg/provider/aws/auth/auth.go b/pkg/provider/aws/auth/auth.go
index 1ba74fcd471..374e2846030 100644
--- a/pkg/provider/aws/auth/auth.go
+++ b/pkg/provider/aws/auth/auth.go
@@ -195,20 +195,23 @@ func NewGeneratorSession(ctx context.Context, auth esv1.AWSAuth, role, region st
 			return nil, err
 		}
 	}
-	config := aws.NewConfig()
+	awscfg, err := config.LoadDefaultConfig(ctx)
+	if err != nil {
+		return nil, err
+	}
 	if credsProvider != nil {
-		config.Credentials = credsProvider
+		awscfg.Credentials = credsProvider
 	}
 	if region != "" {
-		config.Region = region
+		awscfg.Region = region
 	}
 
 	if role != "" {
-		stsclient := assumeRoler(*config)
-		config.Credentials = stscreds.NewAssumeRoleProvider(stsclient, role)
+		stsclient := assumeRoler(awscfg)
+		awscfg.Credentials = stscreds.NewAssumeRoleProvider(stsclient, role)
 	}
-	log.Info("using aws config", "region", config.Region, "credentials", config.Credentials)
-	return config, nil
+	log.Info("using aws config", "region", awscfg.Region, "credentials", awscfg.Credentials)
+	return &awscfg, nil
 }
 
 // credsFromSecretRef pulls access-key / secret-access-key from a secretRef to


Test patch:
diff --git a/pkg/generator/ecr/ecr_test.go b/pkg/generator/ecr/ecr_test.go
index 1a476b8e062..b73054f8eab 100644
--- a/pkg/generator/ecr/ecr_test.go
+++ b/pkg/generator/ecr/ecr_test.go
@@ -54,6 +54,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "nil spec",
 			args: args{
+				ctx: context.Background(),
 				jsonSpec: nil,
 			},
 			wantErr: true,
@@ -61,6 +62,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "invalid json",
 			args: args{
+				ctx: context.Background(),
 				authTokenPrivateFunc: func(gati *ecr.GetAuthorizationTokenInput) (*ecr.GetAuthorizationTokenOutput, error) {
 					return nil, errors.New("boom")
 				},
@@ -73,6 +75,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "private ECR full spec",
 			args: args{
+				ctx: context.Background(),
 				namespace: "foobar",
 				kube: clientfake.NewClientBuilder().WithObjects(&v1.Secret{
 					ObjectMeta: metav1.ObjectMeta{
@@ -123,6 +126,7 @@ spec:
 		{
 			name: "public ECR full spec",
 			args: args{
+				ctx: context.Background(),
 				namespace: "foobar",
 				authTokenPublicFunc: func(in *ecrpublic.GetAuthorizationTokenInput) (*ecrpublic.GetAuthorizationTokenOutput, error) {
 					t := time.Unix(5678, 0)
diff --git a/pkg/generator/sts/sts_test.go b/pkg/generator/sts/sts_test.go
index 4ccf319a72a..6606bd5f792 100644
--- a/pkg/generator/sts/sts_test.go
+++ b/pkg/generator/sts/sts_test.go
@@ -51,6 +51,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "nil spec",
 			args: args{
+				ctx: context.Background(),
 				jsonSpec: nil,
 			},
 			wantErr: true,
@@ -58,6 +59,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "invalid json",
 			args: args{
+				ctx: context.Background(),
 				tokenFunc: func(ctx context.Context, input *sts.GetSessionTokenInput, optFns ...func(*sts.Options)) (*sts.GetSessionTokenOutput, error) {
 					return nil, errors.New("boom")
 				},
@@ -70,6 +72,7 @@ func TestGenerate(t *testing.T) {
 		{
 			name: "full spec",
 			args: args{
+				ctx: context.Background(),
 				namespace: "foobar",
 				kube: clientfake.NewClientBuilder().WithObjects(&v1.Secret{
 					ObjectMeta: metav1.ObjectMeta{
diff --git a/pkg/provider/aws/auth/auth_test.go b/pkg/provider/aws/auth/auth_test.go
index 2ce4a492d2a..90fcc126084 100644
--- a/pkg/provider/aws/auth/auth_test.go
+++ b/pkg/provider/aws/auth/auth_test.go
@@ -647,3 +647,174 @@ func ErrorContains(out error, want string) bool {
 	}
 	return strings.Contains(out.Error(), want)
 }
+
+func TestNewGeneratorSession_DefaultCredentialChain(t *testing.T) {
+	cfg, err := NewGeneratorSession(context.Background(), esv1.AWSAuth{}, "", "us-east-1", clientfake.NewClientBuilder().Build(), "test-ns", DefaultSTSProvider, DefaultJWTProvider)
+	assert.NoError(t, err)
+	assert.NotNil(t, cfg)
+	assert.Equal(t, "us-east-1", cfg.Region)
+}
+
+func TestNewGeneratorSession_CredentialProviderPriority(t *testing.T) {
+	ctx := context.Background()
+	k8sClient := clientfake.NewClientBuilder().Build()
+
+	assert.NoError(t, k8sClient.Create(ctx, &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{Name: "aws-creds", Namespace: "test-ns"},
+		Data: map[string][]byte{
+			"access-key": []byte("SECRET_KEY_ID"),
+			"secret-key": []byte("SECRET_ACCESS_KEY"),
+		},
+	}))
+
+	assert.NoError(t, k8sClient.Create(ctx, &v1.ServiceAccount{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:        "test-sa",
+			Namespace:   "test-ns",
+			Annotations: map[string]string{roleARNAnnotation: "arn:aws:iam::123456789012:role/test-role"},
+		},
+	}))
+
+	jwtProviderCalled := false
+	cfg, err := NewGeneratorSession(ctx, esv1.AWSAuth{
+		JWTAuth: &esv1.AWSJWTAuth{
+			ServiceAccountRef: &esmeta.ServiceAccountSelector{Name: "test-sa"},
+		},
+		SecretRef: &esv1.AWSAuthSecretRef{
+			AccessKeyID:     esmeta.SecretKeySelector{Name: "aws-creds", Key: "access-key"},
+			SecretAccessKey: esmeta.SecretKeySelector{Name: "aws-creds", Key: "secret-key"},
+		},
+	}, "", "us-east-1", k8sClient, "test-ns", DefaultSTSProvider, func(name, namespace, roleArn string, aud []string, region string) (aws.CredentialsProvider, error) {
+		jwtProviderCalled = true
+		assert.Equal(t, "test-sa", name)
+		assert.Equal(t, "test-ns", namespace)
+		assert.Equal(t, "arn:aws:iam::123456789012:role/test-role", roleArn)
+		return fakesess.CredentialsProvider{
+			RetrieveFunc: func() (aws.Credentials, error) {
+				return aws.Credentials{
+					AccessKeyID:     "JWT_ACCESS_KEY",
+					SecretAccessKey: "JWT_SECRET_KEY",
+					SessionToken:    "JWT_SESSION_TOKEN",
+					Source:          "jwt",
+				}, nil
+			},
+		}, nil
+	})
+
+	assert.NoError(t, err)
+	assert.NotNil(t, cfg)
+	assert.True(t, jwtProviderCalled)
+
+	creds, err := cfg.Credentials.Retrieve(ctx)
+	assert.NoError(t, err)
+	assert.Equal(t, "SECRET_KEY_ID", creds.AccessKeyID)
+	assert.Equal(t, "SECRET_ACCESS_KEY", creds.SecretAccessKey)
+}
+
+func TestNewGeneratorSession_OnlySecretRef(t *testing.T) {
+	ctx := context.Background()
+	k8sClient := clientfake.NewClientBuilder().Build()
+
+	assert.NoError(t, k8sClient.Create(ctx, &v1.Secret{
+		ObjectMeta: metav1.ObjectMeta{Name: "aws-creds", Namespace: "test-ns"},
+		Data: map[string][]byte{
+			"access-key": []byte("SECRET_KEY_ID"),
+			"secret-key": []byte("SECRET_ACCESS_KEY"),
+		},
+	}))
+
+	cfg, err := NewGeneratorSession(ctx, esv1.AWSAuth{
+		SecretRef: &esv1.AWSAuthSecretRef{
+			AccessKeyID:     esmeta.SecretKeySelector{Name: "aws-creds", Key: "access-key"},
+			SecretAccessKey: esmeta.SecretKeySelector{Name: "aws-creds", Key: "secret-key"},
+		},
+	}, "", "us-east-1", k8sClient, "test-ns", DefaultSTSProvider, DefaultJWTProvider)
+
+	assert.NoError(t, err)
+	assert.NotNil(t, cfg)
+
+	creds, err := cfg.Credentials.Retrieve(ctx)
+	assert.NoError(t, err)
+	assert.Equal(t, "SECRET_KEY_ID", creds.AccessKeyID)
+	assert.Equal(t, "SECRET_ACCESS_KEY", creds.SecretAccessKey)
+}
+
+func TestNewGeneratorSession_RegionConfiguration(t *testing.T) {
+	ctx := context.Background()
+	k8sClient := clientfake.NewClientBuilder().Build()
+
+	testCases := []struct {
+		name           string
+		region         string
+		expectedRegion string
+	}{
+		{"region specified", "us-east-1", "us-east-1"},
+		{"empty region uses default", "", ""},
+	}
+
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			cfg, err := NewGeneratorSession(ctx, esv1.AWSAuth{}, "", tc.region, k8sClient, "test-ns", DefaultSTSProvider, DefaultJWTProvider)
+			assert.NoError(t, err)
+			assert.NotNil(t, cfg)
+			if tc.expectedRegion != "" {
+				assert.Equal(t, tc.expectedRegion, cfg.Region)
+			}
+		})
+	}
+}
+
+func TestNewGeneratorSession_AssumeRoleWithDefaultCredentials(t *testing.T) {
+	t.Setenv("AWS_ACCESS_KEY_ID", "BASE_ACCESS_KEY")
+	t.Setenv("AWS_SECRET_ACCESS_KEY", "BASE_SECRET_KEY")
+
+	stsProviderCalled := false
+	cfg, err := NewGeneratorSession(context.Background(), esv1.AWSAuth{}, "arn:aws:iam::123456789012:role/assumed-role", "us-east-1", clientfake.NewClientBuilder().Build(), "test-ns", func(cfg aws.Config) STSprovider {
+		stsProviderCalled = true
+		creds, err := cfg.Credentials.Retrieve(context.Background())
+		assert.NoError(t, err)
+		assert.Equal(t, "BASE_ACCESS_KEY", creds.AccessKeyID)
+		return &fakesess.AssumeRoler{
+			AssumeRoleFunc: func(input *sts.AssumeRoleInput) (*sts.AssumeRoleOutput, error) {
+				assert.Equal(t, "arn:aws:iam::123456789012:role/assumed-role", *input.RoleArn)
+				return &sts.AssumeRoleOutput{
+					AssumedRoleUser: &ststypes.AssumedRoleUser{
+						Arn:           aws.String("arn:aws:sts::123456789012:assumed-role/assumed-role/session"),
+						AssumedRoleId: aws.String("AROA123456"),
+					},
+					Credentials: &ststypes.Credentials{
+						AccessKeyId:     aws.String("ASSUMED_ACCESS_KEY"),
+						SecretAccessKey: aws.String("ASSUMED_SECRET_KEY"),
+						SessionToken:    aws.String("ASSUMED_SESSION_TOKEN"),
+						Expiration:      aws.Time(time.Now().Add(time.Hour)),
+					},
+				}, nil
+			},
+		}
+	}, DefaultJWTProvider)
+
+	assert.NoError(t, err)
+	assert.NotNil(t, cfg)
+	assert.True(t, stsProviderCalled)
+
+	creds, err := cfg.Credentials.Retrieve(context.Background())
+	assert.NoError(t, err)
+	assert.Equal(t, "ASSUMED_ACCESS_KEY", creds.AccessKeyID)
+	assert.Equal(t, "ASSUMED_SECRET_KEY", creds.SecretAccessKey)
+	assert.Equal(t, "ASSUMED_SESSION_TOKEN", creds.SessionToken)
+}
+
+func TestNewGeneratorSession_DefaultCredentialChainFallback(t *testing.T) {
+	t.Setenv("AWS_ACCESS_KEY_ID", "ENV_ACCESS_KEY")
+	t.Setenv("AWS_SECRET_ACCESS_KEY", "ENV_SECRET_KEY")
+	t.Setenv("AWS_SESSION_TOKEN", "ENV_SESSION_TOKEN")
+
+	cfg, err := NewGeneratorSession(context.Background(), esv1.AWSAuth{}, "", "us-east-1", clientfake.NewClientBuilder().Build(), "test-ns", DefaultSTSProvider, DefaultJWTProvider)
+	assert.NoError(t, err)
+	assert.NotNil(t, cfg)
+
+	creds, err := cfg.Credentials.Retrieve(context.Background())
+	assert.NoError(t, err)
+	assert.NotEmpty(t, creds.AccessKeyID)
+	assert.NotEmpty(t, creds.SecretAccessKey)
+}