Repository: gruntwork-io/terragrunt (language: go)

Issue:
Could not parse output from terragrunt config when AWS_CSM_ENABLED=true
Hi, I've encountered an error when trying to run `terragrunt apply` with `AWS_CSM_ENABLED` environment variable set to `true`. I receive an error message:

```bash
> cd /the/path/to/terragrunt-projects/project-that-depends-on-cicd-setup
> terragrunt apply
ERRO[0002] Could not parse output from terragrunt config /the/path/to/terragrunt-projects/cicd-setup/terragrunt.hcl. Underlying error: invalid character '/' after top-level value 
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1
```  

However I don't encounter the error when I set:

```bash
# /bin/bash
> export AWS_CSM_ENABLED=false
> terragrunt apply  # provides plan with prompt to apply changes as expected
```

>NOTE: I ran `terragrunt apply --terragrunt-log-level debug --terragrunt-debug`, which provided some additional insight, which gave me the idea to try disabling AWS client side monitoring:

```bash
> terragrunt apply --terragrunt-log-level debug --terragrunt-debug
# -- snip --
DEBU[0000] run_cmd, cached output: [REDACTED]            prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0000] Evaluated 1 locals (remaining 2): git_repo_name  prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0000] Evaluated 1 locals (remaining 1): backend_bucket  prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0000] Evaluated 1 locals (remaining 0): dynamodb_table  prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0000] [Partial] Included config /the/path/to/terragrunt-projects/cicd-setup/terragrunt.hcl has strategy shallow merge: merging config in (shallow).  prefix=[/the/path/to/terragrunt-projects/cicd-setup/cicd-setup] 
DEBU[0000] Detected module /the/path/to/terragrunt-projects/cicd-setup/terragrunt.hcl is already init-ed. Retrieving outputs directly from working directory.  prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0000] Running command: terraform output -json       prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
DEBU[0002] Retrieved output from /the/path/to/terragrunt-projects/cicd-setup/terragrunt.hcl as json: 2022/08/11 08:53:48 Enabling CSM
{
  "codebuild_projects": {
    "sensitive": false,
    "type": [
      "object",
      {
        "pr-build": [
          "object",
          {
            "policy_arn": "string",
            "project_arn": "string",
            "project_id": "string",
            "project_name": "string",
            "project_region": "string",
            "role_arn": "string",
            "role_id": "string"
          }
        ],
        "pr-deploy": [
          "object",
          {
            "policy_arn": "string",
            "project_arn": "string",
            "project_id": "string",
            "project_name": "string",
            "project_region": "string",
            "role_arn": "string",
            "role_id": "string"
          }
        ],
# -- snip --
} prefix=[/the/path/to/terragrunt-projects/cicd-setup] 
ERRO[0002] Could not parse output from terragrunt config /the/path/to/terragrunt-projects/cicd-setup/terragrunt.hcl. Underlying error: invalid character '/' after top-level value 
ERRO[0002] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 
```

I was wondering whether anyone had encountered this issue before and what can be done about it - Fundamentally it seems like a bug in terragrunt,  but you can never be sure with all the moving parts these days 😉 

As a user it would be helpful to have a more specific error for this case (or some additional text stating have I tried disabling AWS CSM) otherwise it is hard to understand the underlying root cause of the error.



Discussion / hints:
For additional context, my `terragrunt.hcl` (within `/the/path/to/terragrunt-projects/project-that-depends-on-cicd-setup`) looks like the following:

```terraform
include "root" {
    path = "${get_repo_root()}/src/terraform/terragrunt.hcl"
}

dependencies {
  paths = ["../cicd-setup"]
}

dependency "cicd" {
  config_path = "../cicd-setup"
}

terraform {
  # NOTE: Try disabling AWS client side monitoring, e.g. `export AWS_CSM_ENABLED=false` if you encounter the following error when running `terragrunt (plan|apply)`:
  #       Could not parse output from terragrunt config python-mono/src/terraform/bootstrap/cicd-setup/terragrunt.hcl. Underlying error: invalid character '/' after top-level value

  before_hook "check_github_token_set" {
    commands     = ["apply", "plan", "destroy"]
    execute      = ["bash", "-c", "if [[ -z $GITHUB_TOKEN ]]; then echo -e '*** \\033[1;33m$GITHUB_TOKEN\\033[0m must be set ***' && exit 126; fi "]
    run_on_error = false
  }
}

inputs = {
  github_actions_codebuild_build_stage_project_name = dependency.cicd.outputs.codebuild_projects["pr-build"].project_name
  github_actions_codebuild_deploy_stage_project_name = dependency.cicd.outputs.codebuild_projects["pr-deploy"].project_name
  github_actions_codebuild_region = dependency.cicd.outputs.codebuild_region
}

```
I'm seeing this too, I tried upgrading to the latest Terragrunt, Terraform and AWS provider versions (details below) but the problem persists whenever I have CSM mode enabled.

In my case the error only seems to manifest when I run Terragrunt in a folder whose `terragrunt.hcl` file declares a dependency block (much like the `cicd` example given by @LiamDeaconAntarcticaAM above, if I've understood it correctly).

Terragrunt: v0.39.1
Terraform: v1.3.2
AWS Provider: v4.35.0
I am seeing this too and was very obscure to track down. 

Error also only manifests when there is a dependency block. 

Terragrunt v0.43.2
Terraform v1.3.9
AWS Provider: 4.49.0

Fix patch:
diff --git a/config/dependency.go b/config/dependency.go
index ee882429b1..c4d7e737ef 100644
--- a/config/dependency.go
+++ b/config/dependency.go
@@ -50,10 +50,11 @@ type Dependency struct {
 
 // DeepMerge will deep merge two Dependency configs, updating the target. Deep merge for Dependency configs is defined
 // as follows:
-// - For simple attributes (bools and strings), the source will override the target.
-// - For MockOutputs, the two maps will be deeply merged together. This means that maps are recursively merged, while
-//   lists are concatenated together.
-// - For MockOutputsAllowedTerraformCommands, the source will be concatenated to the target.
+//   - For simple attributes (bools and strings), the source will override the target.
+//   - For MockOutputs, the two maps will be deeply merged together. This means that maps are recursively merged, while
+//     lists are concatenated together.
+//   - For MockOutputsAllowedTerraformCommands, the source will be concatenated to the target.
+//
 // Note that RenderedOutputs is ignored in the deep merge operation.
 func (targetDepConfig *Dependency) DeepMerge(sourceDepConfig Dependency) error {
 	if sourceDepConfig.ConfigPath != "" {
@@ -145,7 +146,8 @@ var outputLocks = sync.Map{}
 // resulting map as a cty.Value object.
 // TODO: In the future, consider allowing importing dependency blocks from included config
 // NOTE FOR MAINTAINER: When implementing importation of other config blocks (e.g referencing inputs), carefully
-//                      consider whether or not the implementation of the cyclic dependency detection still makes sense.
+//
+//	consider whether or not the implementation of the cyclic dependency detection still makes sense.
 func decodeAndRetrieveOutputs(
 	file *hcl.File,
 	filename string,
@@ -262,8 +264,9 @@ func getDependencyBlockConfigPathsByFilepath(configPath string, terragruntOption
 
 // Encode the list of dependency blocks into a single cty.Value object that maps the dependency block name to the
 // encoded dependency mapping. The encoded dependency mapping should have the attributes:
-// - outputs: The map of outputs of the corresponding terraform module that lives at the target config of the
-//            dependency.
+//   - outputs: The map of outputs of the corresponding terraform module that lives at the target config of the
+//     dependency.
+//
 // This routine will go through the process of obtaining the outputs using `terragrunt output` from the target config.
 func dependencyBlocksToCtyValue(dependencyConfigs []Dependency, terragruntOptions *options.TerragruntOptions) (*cty.Value, error) {
 	paths := []string{}
@@ -322,9 +325,9 @@ func dependencyBlocksToCtyValue(dependencyConfigs []Dependency, terragruntOption
 
 // This will attempt to get the outputs from the target terragrunt config if it is applied. If it is not applied, the
 // behavior is different depending on the configuration of the dependency:
-// - If the dependency block indicates a mock_outputs attribute, this will return that.
-//   If the dependency block indicates a mock_outputs_merge_strategy_with_state attribute, mock_outputs and state outputs will be merged following the merge strategy
-// - If the dependency block does NOT indicate a mock_outputs attribute, this will return an error.
+//   - If the dependency block indicates a mock_outputs attribute, this will return that.
+//     If the dependency block indicates a mock_outputs_merge_strategy_with_state attribute, mock_outputs and state outputs will be merged following the merge strategy
+//   - If the dependency block does NOT indicate a mock_outputs attribute, this will return an error.
 func getTerragruntOutputIfAppliedElseConfiguredDefault(dependencyConfig Dependency, terragruntOptions *options.TerragruntOptions) (*cty.Value, error) {
 	if dependencyConfig.shouldGetOutputs() {
 		outputVal, isEmpty, err := getTerragruntOutput(dependencyConfig, terragruntOptions)
@@ -452,6 +455,14 @@ func getOutputJsonWithCaching(targetConfig string, terragruntOptions *options.Te
 	if err != nil {
 		return nil, err
 	}
+
+	// When AWS Client Side Monitoring (CSM) is enabled the aws-sdk-go displays log as a plaintext "Enabling CSM" to stdout, even if the `output -json` flag is specified. The final output looks like this: "2023/05/04 20:22:43 Enabling CSM{...omitted json string...}", and and prevents proper json parsing. Since there is no way to disable this log, the only way out is to filter.
+	// Related AWS code: https://github.com/aws/aws-sdk-go/blob/81d1cbbc6a2028023aff7bcab0fe1be320cd39f7/aws/session/session.go#L444
+	// Related issues: https://github.com/gruntwork-io/terragrunt/issues/2233 https://github.com/hashicorp/terraform-provider-aws/issues/23620
+	if index := bytes.IndexByte(newJsonBytes, byte('{')); index > 0 {
+		newJsonBytes = newJsonBytes[index:]
+	}
+
 	jsonOutputCache.Store(targetConfig, newJsonBytes)
 	return newJsonBytes, nil
 }
@@ -820,6 +831,7 @@ func terraformOutputJsonToCtyValueMap(targetConfig string, jsonBytes []byte) (ma
 		Value     json.RawMessage `json:"value"`
 	}
 	var outputs map[string]OutputMeta
+
 	err := json.Unmarshal(jsonBytes, &outputs)
 	if err != nil {
 		return nil, errors.WithStackTrace(TerragruntOutputParsingError{Path: targetConfig, Err: err})


Test patch:
diff --git a/test/fixture-output-from-dependency/app/main.tf b/test/fixture-output-from-dependency/app/main.tf
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/test/fixture-output-from-dependency/app/terragrunt.hcl b/test/fixture-output-from-dependency/app/terragrunt.hcl
new file mode 100644
index 0000000000..aa3cce97b8
--- /dev/null
+++ b/test/fixture-output-from-dependency/app/terragrunt.hcl
@@ -0,0 +1,11 @@
+dependencies {
+  paths = ["../dependency"]
+}
+
+dependency "test" {
+  config_path = "../dependency"
+}
+
+inputs = {
+  vpc_config = dependency.test.outputs
+}
diff --git a/test/fixture-output-from-dependency/dependency/main.tf b/test/fixture-output-from-dependency/dependency/main.tf
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/test/fixture-output-from-dependency/dependency/outputs.tf b/test/fixture-output-from-dependency/dependency/outputs.tf
new file mode 100644
index 0000000000..282718fdcd
--- /dev/null
+++ b/test/fixture-output-from-dependency/dependency/outputs.tf
@@ -0,0 +1,4 @@
+output "foo" {
+  value       = var.foo
+  description = "Test foo value"
+}
diff --git a/test/fixture-output-from-dependency/dependency/terragrunt.hcl b/test/fixture-output-from-dependency/dependency/terragrunt.hcl
new file mode 100644
index 0000000000..c14c4a41c0
--- /dev/null
+++ b/test/fixture-output-from-dependency/dependency/terragrunt.hcl
@@ -0,0 +1,16 @@
+# Configure Terragrunt to automatically store tfstate files in an S3 bucket
+remote_state {
+  backend = "s3"
+
+  generate = {
+    path      = "backend.tf"
+    if_exists = "overwrite"
+  }
+
+  config = {
+    encrypt = true
+    bucket  = "__FILL_IN_BUCKET_NAME__"
+    key     = "${path_relative_to_include()}/terraform.tfstate"
+    region  = "__FILL_IN_REGION__"
+  }
+}
diff --git a/test/fixture-output-from-dependency/dependency/variables.tf b/test/fixture-output-from-dependency/dependency/variables.tf
new file mode 100644
index 0000000000..7a16c954e6
--- /dev/null
+++ b/test/fixture-output-from-dependency/dependency/variables.tf
@@ -0,0 +1,5 @@
+variable "foo" {
+  description = "The value to be returned from the module"
+  type        = string
+  default     = "test-foo-value"
+}
diff --git a/test/integration_test.go b/test/integration_test.go
index eef37706cd..9808adfdf0 100644
--- a/test/integration_test.go
+++ b/test/integration_test.go
@@ -59,6 +59,7 @@ const (
 	TEST_FIXTURE_GRAPH_DEPENDENCIES                         = "fixture-graph-dependencies"
 	TEST_FIXTURE_OUTPUT_ALL                                 = "fixture-output-all"
 	TEST_FIXTURE_OUTPUT_FROM_REMOTE_STATE                   = "fixture-output-from-remote-state"
+	TEST_FIXTURE_OUTPUT_FROM_DEPENDENCY                     = "fixture-output-from-dependency"
 	TEST_FIXTURE_STDOUT                                     = "fixture-download/stdout-test"
 	TEST_FIXTURE_EXTRA_ARGS_PATH                            = "fixture-extra-args/"
 	TEST_FIXTURE_ENV_VARS_BLOCK_PATH                        = "fixture-env-vars-block/"
@@ -873,6 +874,34 @@ func TestTerragruntOutputFromRemoteState(t *testing.T) {
 
 	assert.True(t, (strings.Index(output, "app3 output") < strings.Index(output, "app1 output")) && (strings.Index(output, "app1 output") < strings.Index(output, "app2 output")))
 }
+
+func TestTerragruntOutputFromDependency(t *testing.T) {
+	// t.Parallel() cannot be used together with t.Setenv()
+
+	s3BucketName := fmt.Sprintf("terragrunt-test-bucket-%s", strings.ToLower(uniqueId()))
+	defer deleteS3Bucket(t, TERRAFORM_REMOTE_STATE_S3_REGION, s3BucketName)
+
+	tmpEnvPath := copyEnvironment(t, TEST_FIXTURE_OUTPUT_FROM_DEPENDENCY)
+
+	rootTerragruntPath := util.JoinPath(tmpEnvPath, TEST_FIXTURE_OUTPUT_FROM_DEPENDENCY)
+	depTerragruntConfigPath := util.JoinPath(rootTerragruntPath, "dependency", config.DefaultTerragruntConfigPath)
+
+	copyTerragruntConfigAndFillPlaceholders(t, depTerragruntConfigPath, depTerragruntConfigPath, s3BucketName, "not-used", TERRAFORM_REMOTE_STATE_S3_REGION)
+
+	var (
+		stdout bytes.Buffer
+		stderr bytes.Buffer
+	)
+
+	t.Setenv("AWS_CSM_ENABLED", "true")
+
+	err := runTerragruntCommand(t, fmt.Sprintf("terragrunt run-all apply -auto-approve --terragrunt-non-interactive --terragrunt-working-dir %s --terragrunt-log-level debug", rootTerragruntPath), &stdout, &stderr)
+	assert.NoError(t, err)
+
+	output := stderr.String()
+	assert.NotContains(t, output, "invalid character")
+}
+
 func TestTerragruntValidateAllCommand(t *testing.T) {
 	t.Parallel()