Repository: gruntwork-io/terragrunt

Issue:
AWS SSO error SSOProviderInvalidToken: the SSO session has expired or is invalid
**Describe the bug**
After logging using aws sso I am able to run aws cli command, deploy terraform modules, however I receive errors related to an invalid session if I try to use Terragrunt

**To Reproduce**
1. Log in using aws sso: `aws sso login --profile <profile name>`
1. List your bucket: `aws s3 ls`
1. Run a plan on a given **Terraform** module
1. Try to run a plan with **Terragrunt** 

**Expected behavior**
Terragrunt commands should be able to run as the we are already authenticated.

- [x] Terminal output (some directories have been sanitized):
```bash
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region
DEBU[0000] [Partial] Included config <path to project>/us-east-1/region_common.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): dynamodb_table, iam_role, aws_account_id, terraform_state_s3_bucket, terraform_state_aws_region  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): iam_role, aws_account_id, terraform_state_s3_bucket, terraform_state_aws_region, dynamodb_table  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 1 locals (remaining 0): deployment_commons
DEBU[0000] [Partial] Included config <path to project>/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Running command: terraform --version          prefix=[<path to project>/us-east-1/vpc]
DEBU[0000] Terraform version: 1.4.6
DEBU[0000] Reading Terragrunt config file at <path to project>/us-east-1/vpc/terragrunt.hcl
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region
DEBU[0000] [Partial] Included config <path to project>/us-east-1/region_common.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): terraform_state_aws_region, dynamodb_table, iam_role, aws_account_id, terraform_state_s3_bucket  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 1 locals (remaining 0): deployment_commons
DEBU[0000] [Partial] Included config <path to project>/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Did not find any locals block: skipping evaluation.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region
DEBU[0000] Included config <path to project>/us-east-1/region_common.hcl has strategy shallow merge: merging config in (shallow) for dependency.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): aws_account_id, terraform_state_s3_bucket, terraform_state_aws_region, dynamodb_table, iam_role  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 1 locals (remaining 0): deployment_commons
DEBU[0000] Included config <path to project>/terragrunt.hcl has strategy shallow merge: merging config in (shallow) for dependency.
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Evaluated 1 locals (remaining 0): aws_region
DEBU[0000] Included config <path to project>/us-east-1/region_common.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): terraform_state_s3_bucket, terraform_state_aws_region, dynamodb_table, iam_role, aws_account_id  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 1 locals (remaining 0): deployment_commons
DEBU[0000] Found locals block: evaluating the expressions.
DEBU[0000] Found locals block: evaluating the expressions.  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 5 locals (remaining 0): terraform_state_s3_bucket, terraform_state_aws_region, dynamodb_table, iam_role, aws_account_id  prefix=[<Home dir>/repos/IaC]
DEBU[0000] Evaluated 1 locals (remaining 0): deployment_commons
DEBU[0000] Included config <path to project>/terragrunt.hcl has strategy shallow merge: merging config in (shallow).
DEBU[0000] Terraform files in <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example are up to date. Will not download again.
DEBU[0000] Copying files from <path to project>/us-east-1/vpc into <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example
DEBU[0000] Setting working directory to <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example
DEBU[0000] The file path <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example/provider.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.  prefix=[<path to project>/us-east-1/vpc]
DEBU[0000] Generated file <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example/provider.tf.  prefix=[<path to project>/us-east-1/vpc]
DEBU[0000] The file path <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example/backend.tf already exists, but was a previously generated file by terragrunt. Since if_exists for code generation is set to "overwrite_terragrunt", regenerating file.  prefix=[<path to project>/us-east-1/vpc]
DEBU[0000] Generated file <cache dir>/terragrunt-cache/_jqBrbAj4YhJ70GhGygJ049fXow/1DCsnRS7PvRws6ncZHCJ4Xq1oDQ/modules/networking/vpc-example/backend.tf.  prefix=[<path to project>/us-east-1/vpc]
DEBU[0000] Initializing remote state for the s3 backend  prefix=[<path to project>/us-east-1/vpc]
ERRO[0000] Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): SSOProviderInvalidToken: the SSO session has expired or is invalid
caused by: open <Home dir>/.aws/sso/cache/64041c5d376abe231cc23768caf97288e7e11277.json: no such file or directory
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1
```
- [x] Screenshots:
![image](https://github.com/gruntwork-io/terragrunt/assets/44711170/7217d035-8968-4b04-9127-cab85e32d9f8)


**Versions**
- Terragrunt version: v0.46.3, I also tried version v0.45.11 and the result is the same.
- Terraform version: v1.4.6
- Environment details (Ubuntu 20.04, RHEL 8.):

**Additional context**
Add any other context about the problem here.

Hints from issue comments:
@yiskaneto: **Update 1 (Preferred workaround for now)**

I was able to make it to work by configuring AWS SSO without a session name, so when running `aws configure sso` don't provide a name for the session when the  `SSO session name (Recommended):` input message is prompted, just hit enter, this will return:
```bash
WARNING: Configuring using legacy format (e.g. without an SSO session).
Consider re-running "configure sso" command and providing a session name.
```
**Note**: When providing the sso_start_url value make sure that it does **NOT** contain the `#` symbol at the end.

Continue with the configuration and now the Terragrunt commands should work@yiskaneto: **Update 2** (Leaving here just for informational purposes but don't use this mehod)

I was able to determine that if the url on the `sso_start_url` value of the profile block on the `~/.aws.configure` file does NOT end with `#/` then,  the  `SSOProviderInvalidToken: the SSO session has expired or is invalid ` error appears when running Terragrunt commands. So configure aws sso as you would normally do assigning a name for the session and the profile but make sure that the urls end with `#/`. Ultimately the `~/.aws.configure` file should look like this

```bash
[profile <profile name>]
sso_session = <profile name>
sso_account_id = <account id>
sso_role_name = <role name>
region = <region>
output = <output>
sso_start_url = https://<url>/start#/
sso_region = <region>

[sso-session ska]
sso_start_url = https://<url>/start#/
sso_region = <region>
sso_registration_scopes =<sso_registration_scopes>

```@RafaelClovr: I can confirm that issue with:
* Terraform: 1.5
* AWS provider: 5.4.0
@gszakonyGW: I can confirm this bug as well.

- Terrragrunt: 0.47.0
- Terraform: 1.5.1
- AWS provider: 5.5.0
Environment details (Apple M1 Pro)
@stnorbi: Hi, 

Is there any ongoing activity resolve this issue? I have been planning to use this problematic approach in a corporate project.

Thank you for accelerate the resolution.

Cheers!@ipcechinel: I can confirm this bug as well.

Terrragrunt: 0.48.0
Terraform: 1.5.2
AWS provider: 5.5.0
Environment details (Apple M2 Pro)@gszakonyGW: I have to add that I can only replicate the bug on my home network as I don't see the issue if I am in the office. So I wonder about root cause. @ajax-bychenok-y: Workaround by @escanoru works for me. Thank you!@svyotov: Same issue for me, the work around does not work.
 Terraform: 1.5.3@yiskaneto: > Same issue for me, the work around does not work. Terraform: 1.5.3

I updated Terraform to 1.5.3 and Terragrunt to 0.48.1 and ran into an into the following issue:
```bash
Error loading SSO Token: Token for https://<URL>/start# does not exist
```

So I updated the workaround 1 steps (https://github.com/gruntwork-io/terragrunt/issues/2604#issuecomment-1588478178) and is working for me now.@mattmccarty: > **Note**: When providing the sso_start_url value make sure that it does **NOT** contain the `#` symbol at the end.

This fixed my issue
@tsbatista: Unfortunately for many of us, using a single profile does not work, hence workaround 1 is flawed as it forces the user to have a nameless sso session.

What I found to be a usable workaround working with multiple accounts (you will loose the auto refresh) is to export the credentials using 
```
aws configure export-credentials --format env
```
and using those in the shell where you will then use terragrunt.
@rpgeddam: FWIW, this doesn't seem to be a terragrunt specific issue. I ran into this issue using just plain ol' terraform v1.5.3 and google brought me to this page. The workaround at the top of the thread works though.@crotger: here's a slightly different workaround that is a little less effort:
```
[profile terraform]
credential_process = aws --profile terraform-sso configure export-credentials --format process
region = <region>
output = json

[profile terraform-sso]
sso_session = <session>
<... rest of SSO config>
```

Configure the `terraform-sso` profile with the SSO session name as usual using `aws configure sso`, then you can point the `terraform` profile to dump the creds from that one using credential_process, and use `AWS_PROFILE=terraform` for actually running terraform. You'll still have to log in with the terraform-sso profile, however.@yiskaneto: > here's a slightly different workaround that is a little less effort:
> 
> ```
> [profile terraform]
> credential_process = aws --profile terraform-sso configure export-credentials --format process
> region = <region>
> output = json
> 
> [profile terraform-sso]
> sso_session = <session>
> <... rest of SSO config>
> ```
> 
> Configure the `terraform-sso` profile with the SSO session name as usual using `aws configure sso`, then you can point the `terraform` profile to dump the creds from that one using credential_process, and use `AWS_PROFILE=terraform` for actually running terraform. You'll still have to log in with the terraform-sso profile, however.

This is the best approach thus far to workaround this when having multiple profiles.@bryceml: Looks like terraform itself is going to start supporting the new format soon: https://github.com/hashicorp/terraform/issues/32465#issuecomment-1713948699

Would be nice if this were fixed in time for that release.@benjefferies: 1.6.0 is released now. It would be good to fix this in terragrunt 🎉 @lev-ok: Resolved in [v0.53.1](https://github.com/gruntwork-io/terragrunt/releases/tag/v0.53.1) release.@yangyucaida: Actually after "aws configure sso", "aws sso login" helps me out finally without error like:
 Error: SSOProviderInvalidToken: the SSO session has expired or is invalid
│ caused by: open C:\Users\yangy\.aws\sso\cache\6ffbea9ac359b6f757ec07da3f5fe310386fb121.json: The system cannot find the file specified.@nezakoo: For those who are still experiencing issues with Terraform and AWS SSO authentication **but cannot upgrade terraform**, I found a solution that worked for me. The problem seems to stem from a mismatch in how the filename hash for the SSO cache is generated between the AWS CLI and Terraform.

**Solution**:
I removed the `sso_session = <masked>` line from my AWS config. This change appears to align the hash generation processes of AWS CLI and Terraform by simplifying the configuration.

**Explanation**:
It seems that the hash used to name the SSO cache file in `~/.aws/sso/cache/` is derived from the details in your AWS configuration. Terraform and AWS CLI were expecting different hashes due to slight differences in how they processed the AWS config details, particularly around how they handle the sso_session parameter.

By removing this parameter, both AWS CLI and Terraform now generate and look for the same filename, which resolves the error related to the SSO session token being invalid or expired.

> I used Terraform v1.5.7@jllg18: your backend was created already? 

if yes you need to use the data to retrieve it. @diphuaji: The same issue will happy for python programs using boto3. Somehow it computes a different hash than the one generated by AWS cli@aviden: > here's a slightly different workaround that is a little less effort:
> 
> ```
> [profile terraform]
> credential_process = aws --profile terraform-sso configure export-credentials --format process
> region = <region>
> output = json
> 
> [profile terraform-sso]
> sso_session = <session>
> <... rest of SSO config>
> ```
> 
> Configure the `terraform-sso` profile with the SSO session name as usual using `aws configure sso`, then you can point the `terraform` profile to dump the creds from that one using credential_process, and use `AWS_PROFILE=terraform` for actually running terraform. You'll still have to log in with the terraform-sso profile, however.

@crotger: worked like a charm, thanks man!

Solution patch:
diff --git a/docs/_docs/04_reference/config-blocks-and-attributes.md b/docs/_docs/04_reference/config-blocks-and-attributes.md
index 4d882a5c..d3745ec9 100644
--- a/docs/_docs/04_reference/config-blocks-and-attributes.md
+++ b/docs/_docs/04_reference/config-blocks-and-attributes.md
@@ -1260,6 +1260,26 @@ terragrunt_version_constraint = ">= 0.23"
 The terragrunt `retryable_errors` list can be used to override the default list of retryable errors with your own custom list.
 To learn more about the `retryable_errors` attribute, see the [auto-retry feature overview](/docs/features/auto-retry).
 
+Default List:
+
+```hcl
+retryable_errors = [
+  "(?s).*Failed to load state.*tcp.*timeout.*",
+  "(?s).*Failed to load backend.*TLS handshake timeout.*",
+  "(?s).*Creating metric alarm failed.*request to update this alarm is in progress.*",
+  "(?s).*Error installing provider.*TLS handshake timeout.*",
+  "(?s).*Error configuring the backend.*TLS handshake timeout.*",
+  "(?s).*Error installing provider.*tcp.*timeout.*",
+  "(?s).*Error installing provider.*tcp.*connection reset by peer.*",
+  "NoSuchBucket: The specified bucket does not exist",
+  "(?s).*Error creating SSM parameter: TooManyUpdates:.*",
+  "(?s).*app.terraform.io.*: 429 Too Many Requests.*",
+  "(?s).*ssh_exchange_identification.*Connection closed by remote host.*",
+  "(?s).*Client\\.Timeout exceeded while awaiting headers.*",
+  "(?s).*Could not download module.*The requested URL returned error: 429.*",
+]
+```
+
 Example:
 
 ```hcl
diff --git a/go.mod b/go.mod
index 1e6c6f57..882e59d8 100644
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,7 @@ go 1.21
 
 require (
 	cloud.google.com/go/storage v1.28.1
-	github.com/aws/aws-sdk-go v1.44.122
+	github.com/aws/aws-sdk-go v1.46.6
 	github.com/creack/pty v1.1.11
 	github.com/fatih/structs v1.1.0
 	github.com/go-errors/errors v1.4.2
diff --git a/go.sum b/go.sum
index cb431ce3..8f369fe7 100644
--- a/go.sum
+++ b/go.sum
@@ -296,8 +296,9 @@ github.com/armon/go-radix v1.0.0/go.mod h1:ufUuZ+zHj4x4TnLV4JWEpy2hxWSpsRywHrMgI
 github.com/aws/aws-sdk-go v1.15.78/go.mod h1:E3/ieXAlvM0XWO57iftYVDLLvQ824smPP3ATZkfNZeM=
 github.com/aws/aws-sdk-go v1.31.9/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/aws/aws-sdk-go v1.37.0/go.mod h1:hcU610XS61/+aQV88ixoOzUoG7v3b31pl2zKMmprdro=
-github.com/aws/aws-sdk-go v1.44.122 h1:p6mw01WBaNpbdP2xrisz5tIkcNwzj/HysobNoaAHjgo=
 github.com/aws/aws-sdk-go v1.44.122/go.mod h1:y4AeaBuwd2Lk+GepC1E9v0qOiTws0MIWAX4oIKwKHZo=
+github.com/aws/aws-sdk-go v1.46.6 h1:6wFnNC9hETIZLMf6SOTN7IcclrOGwp/n9SLp8Pjt6E8=
+github.com/aws/aws-sdk-go v1.46.6/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
 github.com/baiyubin/aliyun-sts-go-sdk v0.0.0-20180326062324-cfa1a18b161f/go.mod h1:AuiFmCCPBSrqvVMvuqFuk0qogytodnVFVSN5CeJB8Gc=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
 github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=



Test patch:
diff --git a/test/integration_test.go b/test/integration_test.go
index bee647fb..56f0bbb2 100644
--- a/test/integration_test.go
+++ b/test/integration_test.go
@@ -1050,13 +1050,16 @@ func TestTerragruntStackCommands(t *testing.T) {
 	defer deleteS3Bucket(t, TERRAFORM_REMOTE_STATE_S3_REGION, s3BucketName)
 	defer cleanupTableForTest(t, lockTableName, TERRAFORM_REMOTE_STATE_S3_REGION)
 
+	cleanupTerraformFolder(t, TEST_FIXTURE_STACK)
+	cleanupTerragruntFolder(t, TEST_FIXTURE_STACK)
+
 	tmpEnvPath := copyEnvironment(t, TEST_FIXTURE_STACK)
 
-	rootTerragruntConfigPath := util.JoinPath(tmpEnvPath, "fixture-stack", config.DefaultTerragruntConfigPath)
+	rootTerragruntConfigPath := util.JoinPath(tmpEnvPath, TEST_FIXTURE_STACK, config.DefaultTerragruntConfigPath)
 	copyTerragruntConfigAndFillPlaceholders(t, rootTerragruntConfigPath, rootTerragruntConfigPath, s3BucketName, lockTableName, "not-used")
 
-	mgmtEnvironmentPath := fmt.Sprintf("%s/fixture-stack/mgmt", tmpEnvPath)
-	stageEnvironmentPath := fmt.Sprintf("%s/fixture-stack/stage", tmpEnvPath)
+	mgmtEnvironmentPath := util.JoinPath(tmpEnvPath, TEST_FIXTURE_STACK, "mgmt")
+	stageEnvironmentPath := util.JoinPath(tmpEnvPath, TEST_FIXTURE_STACK, "stage")
 
 	runTerragrunt(t, fmt.Sprintf("terragrunt apply-all --terragrunt-non-interactive --terragrunt-working-dir %s", mgmtEnvironmentPath))
 	runTerragrunt(t, fmt.Sprintf("terragrunt apply-all --terragrunt-non-interactive --terragrunt-working-dir %s", stageEnvironmentPath))
@@ -4800,7 +4803,8 @@ func TestTerragruntOutputFromRemoteState(t *testing.T) {
 func TestShowErrorWhenRunAllInvokedWithoutArguments(t *testing.T) {
 	t.Parallel()
 
-	appPath := TEST_FIXTURE_STACK
+	tmpEnvPath := copyEnvironment(t, TEST_FIXTURE_STACK)
+	appPath := util.JoinPath(tmpEnvPath, TEST_FIXTURE_STACK)
 
 	stdout := bytes.Buffer{}
 	stderr := bytes.Buffer{}
@@ -5641,6 +5645,7 @@ func TestInitFailureModulePrefix(t *testing.T) {
 	t.Parallel()
 
 	initTestCase := TEST_FIXTURE_INIT_ERROR
+
 	cleanupTerraformFolder(t, initTestCase)
 	cleanupTerragruntFolder(t, initTestCase)
 
@@ -5681,7 +5686,9 @@ func TestDependencyOutputModulePrefix(t *testing.T) {
 func TestErrorExplaining(t *testing.T) {
 	t.Parallel()
 
-	initTestCase := TEST_FIXTURE_INIT_ERROR
+	tmpEnvPath := copyEnvironment(t, TEST_FIXTURE_INIT_ERROR)
+	initTestCase := util.JoinPath(tmpEnvPath, TEST_FIXTURE_INIT_ERROR)
+
 	cleanupTerraformFolder(t, initTestCase)
 	cleanupTerragruntFolder(t, initTestCase)