Repository: gruntwork-io/terragrunt (language: go)

Issue:
Retryable errors do not work when using the -json flag
When using the command `terragrunt apply -auto-approve -json` with retryable errors, retries will never occur. The exact same configuration will perform a retry when the `-json` flag is removed. 

**Reproduction Steps** 

Reproduced on
Terraform: `1.0.8` and `1.39.0`
Terragrunt: `0.36.6` and `0.43.2`

Configure a single `terragrunt.hcl` file
```terraform
terraform {
  source = "../..//modules"
}

retryable_errors = [
  ".*"
]

# Indicate the input values to use for the variables of the module.
inputs = {
    input = "hello world"
    filename = ""
}
```

Define the module with the code below
```terraform
variable "input" {
  type = string

}

variable "filename" {
  type = string
}

variable "file_num" {
  type    = number
  default = 0
}

resource "local_file" "foo" {
  content  = var.input
  filename = var.filename
}

resource "local_file" "foos" {
  count    = 10
  content  = "${var.input}${count.index}"
  filename = "${var.filename}${count.index}"
}

locals {
  f = lookup(one(local_file.foos), "id", "")
}

resource "local_file" "foosum" {
  content  = join(",", local.f)
  filename = "out.txt"
}
```

run `terragrunt apply -auto-approve -json`

**Results**
```
Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Using previously-installed hashicorp/local v2.3.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
{"@level":"info","@message":"Terraform 1.3.9","@module":"terraform.ui","@timestamp":"2023-02-22T08:35:26.927999-05:00","terraform":"1.3.9","type":"version","ui":"1.0"}
{"@level":"error","@message":"Error: Invalid function argument","@module":"terraform.ui","@timestamp":"2023-02-22T08:35:27.215732-05:00","diagnostic":{"severity":"error","summary":"Invalid function argument","detail":"Invalid value for \"list\" parameter: must be a list, set, or tuple value with either zero or one elements.","range":{"filename":"main.tf","start":{"line":18,"column":18,"byte":344},"end":{"line":18,"column":33,"byte":359}},"snippet":{"context":"locals","code":"  f = lookup(one(local_file.foos), \"id\", \"\")","start_line":18,"highlight_start_offset":17,"highlight_end_offset":32,"values":[{"traversal":"local_file.foos","statement":"is tuple with 10 elements"}],"function_call":{"called_as":"one","signature":{"name":"one","params":[{"name":"list","type":"dynamic"}],"return_type":"dynamic"}}}},"type":"diagnostic"}
ERRO[0000] Terraform invocation failed in /Users/username/Documents/GitHub/internal/pytest-terraform/temp_deploy/deployment/dep/.terragrunt-cache/7bBl73k5tubZfPJe-PSv6BKIEyQ/GJj-N-VnhocgWZJslk2PIAA0o2M/modules  prefix=[/Users/username/Documents/GitHub/internal/pytest-terraform/temp_deploy/deployment/dep] 
ERRO[0000] 1 error occurred:
        * exit status 1
```

running the same setup without the `-json`  
`terragrunt apply -auto-approve`
```
Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Using previously-installed hashicorp/local v2.3.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
╷
│ Error: Invalid function argument
│ 
│   on main.tf line 30, in locals:
│   30:   f = lookup(one(local_file.foos), "id", "")
│     ├────────────────
│     │ while calling one(list)
│     │ local_file.foos is tuple with 10 elements
│ 
│ Invalid value for "list" parameter: must be a list, set, or tuple value
│ with either zero or one elements.
╵
INFO[0001] Encountered an error eligible for retrying. Sleeping 5s before retrying.  prefix=[/Users/username/Documents/GitHub/internal/pytest-terraform/temp_deploy/deployment/dep] 
╷
│ Error: Invalid function argument
│ 
│   on main.tf line 30, in locals:
│   30:   f = lookup(one(local_file.foos), "id", "")
│     ├────────────────
│     │ while calling one(list)
│     │ local_file.foos is tuple with 10 elements
│ 
│ Invalid value for "list" parameter: must be a list, set, or tuple value
│ with either zero or one elements.
╵
INFO[0006] Encountered an error eligible for retrying. Sleeping 5s before retrying.  prefix=[/Users/username/Documents/GitHub/internal/pytest-terraform/temp_deploy/deployment/dep] 
╷
│ Error: Invalid function argument
│ 
│   on main.tf line 30, in locals:
│   30:   f = lookup(one(local_file.foos), "id", "")
│     ├────────────────
│     │ while calling one(list)
│     │ local_file.foos is tuple with 10 elements
│ 
│ Invalid value for "list" parameter: must be a list, set, or tuple value
│ with either zero or one elements.
╵
INFO[0011] Encountered an error eligible for retrying. Sleeping 5s before retrying.  prefix=[/Users/username/Documents/GitHub/internal/pytest-terraform/temp_deploy/deployment/dep] 
ERRO[0016] 1 error occurred:
        * Exhausted retries (3) for command terraform apply -auto-approve
 
ERRO[0016] Unable to determine underlying exit code, so Terragrunt will exit with error code 1
```





Discussion / hints:
Debugged terragrunt. I found the root cause of the issue
[Here](https://github.com/gruntwork-io/terragrunt/blob/fbca9a81b0d52f85f1b1c9dbb1620d47b003a910/cli/cli_app.go#L928), the code is getting the `stdout` and `stderr`
`out.Stderr` without -json
```
"\x1b[31m╷\x1b[0m\x1b[0m\n\x1b[31m│\x1b[0m \x1b[0m\x1b[1m\x1b[31mError: \x1b[0m\x1b[0m\x1b[1mInvalid function argument\x1b[0m\n\x1b[31m│\x1b[0m \x1b[0m\n\x1b[31m│\x1b[0m \x1b[0m\x1b[0m  on main.tf line 30, in locals:\n\x1b[31m│\x1b[0m \x1b[0m  30:   f = lookup(one(\x1b[4mlocal_file.foos\x1b[0m), \"id\", \"\")\x1b[0m\n\x1b[31m│\x1b[0m \x1b[0m    \x1b[90m├────────────────\x1b[0m\n\x1b[31m│\x1b[0m \x1b[0m\x1b[0m    \x1b[90m│\x1b[0m while calling \x1b[1mone\x1b[0m(\x1b[0mlist)\n\x1b[31m│\x1b[0m \x1b[0m    \x1b[90m│\x1b[0m \x1b[1mlocal_file.foos\x1b[0m is tuple with 10 elements\n\x1b[31m│\x1b[0m \x1b[0m\x1b[0m\n\x1b[31m│\x1b[0m \x1b[0mInvalid value for \"list\" parameter: must be a list, set, or tuple value\n\x1b[31m│\x1b[0m \x1b[0mwith either zero or one elements.\n\x1b[31m╵\x1b[0m\x1b[0m\n"
```



when the flag `-json` is included. the `out.Stderr` is always an empty string and `out.Stdout` contains the error
```
"{\"@level\":\"info\",\"@message\":\"Terraform 1.3.9\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-02-22T10:46:18.299515-05:00\",\"terraform\":\"1.3.9\",\"type\":\"version\",\"ui\":\"1.0\"}\n{\"@level\":\"error\",\"@message\":\"Error: Invalid function argument\",\"@module\":\"terraform.ui\",\"@timestamp\":\"2023-02-22T10:46:18.709985-05:00\",\"diagnostic\":{\"severity\":\"error\",\"summary\":\"Invalid function argument\",\"detail\":\"Invalid value for \\\"list\\\" parameter: must be a list, set, or tuple value with either zero or one elements.\",\"range\":{\"filename\":\"main.tf\",\"start\":{\"line\":30,\"column\":18,\"byte\":386},\"end\":{\"line\":30,\"column\":33,\"byte\":401}},\"snippet\":{\"context\":\"locals\",\"code\":\"  f = lookup(one(local_file.foos), \\\"id\\\", \\\"\\\")\",\"start_line\":30,\"highlight_start_offset\":17,\"highlight_end_offset\":32,\"values\":[{\"traversal\":\"local_file.foos\",\"statement\":\"is tuple with 10 elements\"}],\"function_call\":{\"called_as\":\"one\",\"signature\":{\"name\":\"one\",\"params\":[{\"name\":\"list\",\"type\":\"dynamic\"}],\"return_type\":\"dynamic\"}}}},\"type\":\"diagnostic\"}\n"
``` 
This means the retryable errors will never find errors when `-json`  is enabled

@leighpascoe Thanks for the detailed bug report and root cause analysis! I'll get this into our backlog for prioritization, but it may be a while before we can get a fix out. If you have the time and are comfortable with the repo a PR would be welcome, otherwise we'll keep you updated here on any progress.

Fix patch:
diff --git a/cli/cli_app.go b/cli/cli_app.go
index fb9bd94061..be33c39777 100644
--- a/cli/cli_app.go
+++ b/cli/cli_app.go
@@ -926,7 +926,7 @@ func runTerraformWithRetry(terragruntOptions *options.TerragruntOptions) error {
 	// Retry the command configurable time with sleep in between
 	for i := 0; i < terragruntOptions.RetryMaxAttempts; i++ {
 		if out, tferr := shell.RunTerraformCommandWithOutput(terragruntOptions, terragruntOptions.TerraformCliArgs...); tferr != nil {
-			if out != nil && isRetryable(out.Stderr, tferr, terragruntOptions) {
+			if out != nil && isRetryable(out.Stdout, out.Stderr, tferr, terragruntOptions) {
 				terragruntOptions.Logger.Infof("Encountered an error eligible for retrying. Sleeping %v before retrying.\n", terragruntOptions.RetrySleepIntervalSec)
 				time.Sleep(terragruntOptions.RetrySleepIntervalSec)
 			} else {
@@ -1203,12 +1203,13 @@ func checkProtectedModule(terragruntOptions *options.TerragruntOptions, terragru
 	return nil
 }
 
-// isRetryable checks whether there was an error and we should attempt again
-func isRetryable(tfoutput string, tferr error, terragruntOptions *options.TerragruntOptions) bool {
+// isRetryable checks whether there was an error and if the output matches any of the configured RetryableErrors
+func isRetryable(stdout string, stderr string, tferr error, terragruntOptions *options.TerragruntOptions) bool {
 	if !terragruntOptions.AutoRetry || tferr == nil {
 		return false
 	}
-	return util.MatchesAny(terragruntOptions.RetryableErrors, tfoutput)
+	// When -json is enabled, Terraform will send all output, errors included, to stdout.
+	return util.MatchesAny(terragruntOptions.RetryableErrors, stderr) || util.MatchesAny(terragruntOptions.RetryableErrors, stdout)
 }
 
 // Custom error types


Test patch:
diff --git a/cli/cli_app_test.go b/cli/cli_app_test.go
index 3dee1b41a1..47e4f5986f 100644
--- a/cli/cli_app_test.go
+++ b/cli/cli_app_test.go
@@ -5,9 +5,9 @@ import (
 	"path/filepath"
 	"testing"
 
-	"github.com/gruntwork-io/terragrunt/errors"
-
+	goerrors "github.com/go-errors/errors"
 	"github.com/gruntwork-io/terragrunt/config"
+	"github.com/gruntwork-io/terragrunt/errors"
 	"github.com/gruntwork-io/terragrunt/options"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -128,6 +128,118 @@ func TestTerragruntTerraformCodeCheck(t *testing.T) {
 	}
 }
 
+func TestErrorRetryableOnStdoutError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{".*error.*"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("error is here", "", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.True(t, retryable, "The error should have retried")
+}
+
+func TestErrorMultipleRetryableOnStderrError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{"no match", ".*error.*"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("", "error is here", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.True(t, retryable, "The error should have retried")
+}
+
+func TestEmptyRetryablesOnStderrError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("", "error is here", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.False(t, retryable, "The error should not have retried, the list of retryable errors was empty")
+}
+
+func TestErrorRetryableOnStderrError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{".*error.*"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("", "error is here", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.True(t, retryable, "The error should have retried")
+}
+
+func TestErrorNotRetryableOnStdoutError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{"not the error"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("error is here", "", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.False(t, retryable, "The error should not retry")
+}
+
+func TestErrorNotRetryableOnStderrError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{"not the error"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("", "error is here", errors.WithStackTrace(goerrors.New("dummy error")), tgOptions)
+	require.False(t, retryable, "The error should not retry")
+}
+
+func TestErrorNotRetryableOnStderrWithoutError(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{".*"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = true
+
+	retryable := isRetryable("", "error is here", nil, tgOptions)
+	require.False(t, retryable, "The error should not retry")
+}
+
+func TestAutoRetryFalseDisablesRetry(t *testing.T) {
+	t.Parallel()
+
+	tgOptions, err := options.NewTerragruntOptionsForTest("")
+	require.NoError(t, err)
+
+	retryableErrors := []string{".*"}
+	tgOptions.RetryableErrors = retryableErrors
+	tgOptions.AutoRetry = false
+
+	retryable := isRetryable("", "error is here", nil, tgOptions)
+	require.False(t, retryable, "The error should not retry")
+}
+
 func TestTerragruntHandlesCatastrophicTerraformFailure(t *testing.T) {
 	t.Parallel()