Repository: Dynatrace/dynatrace-configuration-as-code
Pull request #1810: fix: `\n`, `\t`, `\r` are no longer included in ME ids
Author: login: Laubi
Created: 2025-03-26T15:49:20Z | Merged: 2025-03-28T11:37:58Z


#### What this PR does / Why we need it:
Fixes a small issue where ME IDs accidently included the `n` from `\n` if there were no extra characters in between.

#### Note for reviewers:

The searching is done after extracting ME IDs, as it can't happen with UUIDs.


Diff:
diff --git a/pkg/download/id_extraction/id_extraction.go b/pkg/download/id_extraction/id_extraction.go
index 7d562f1a2..d182ec481 100644
--- a/pkg/download/id_extraction/id_extraction.go
+++ b/pkg/download/id_extraction/id_extraction.go
@@ -95,9 +95,23 @@ func ExtractIDsIntoYAML(configsPerType project.ConfigsPerType) (project.ConfigsP
 	return configsPerType, nil
 }
 
+// invalidMeId finds ME IDs in the form of `nABC`, `rABC`, and `tABC`. They are most commonly mistakenly created by `\nABC`.
+var invalidMeId = regexp.MustCompile("[nrt][A-Z]+")
+
 func findAllIds(content string) []string {
 	ids := meIDRegexPattern.FindAllString(content, -1)
+
+	// Super simple, hacky, sanity check to not accidentally include matches for ME IDs in the form of `nABC-123` which result from `\nABC-123`.
+	// Alternatives to the check is validating the content of matches for `\`, but a `\` can also be escaped and would require recursive checking.
+	// To not do that, we simply check if the id starts with `n`, `r`, `t` (from `\n`, `\r`, `\t`), and then only uppercase characters. This is a good check for potential issues.
+	for i, id := range ids {
+		if invalidMeId.MatchString(id) {
+			ids[i] = id[1:]
+		}
+	}
+
 	ids = append(ids, uuidRegexPattern.FindAllString(content, -1)...)
+
 	return ids
 }
 
diff --git a/pkg/download/id_extraction/id_extraction_test.go b/pkg/download/id_extraction/id_extraction_test.go
index 63ed581d7..f12b004be 100644
--- a/pkg/download/id_extraction/id_extraction_test.go
+++ b/pkg/download/id_extraction/id_extraction_test.go
@@ -21,6 +21,8 @@ package id_extraction
 import (
 	"testing"
 
+	"github.com/google/go-cmp/cmp"
+	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/stretchr/testify/assert"
 
 	"github.com/dynatrace/dynatrace-configuration-as-code/v2/internal/featureflags"
@@ -379,13 +381,37 @@ func TestScopeParameterIsTreatedAsParameter(t *testing.T) {
 				},
 			},
 		},
+		{
+			"scope parameter with environment value is not treated as separate param",
+			project.ConfigsPerType{
+				"test-type": []config.Config{
+					{
+						Template:   template.NewInMemoryTemplate("test-tmpl", `{"key": "value\nCLOUD_APPLICATION-0000000000000000"}`),
+						Parameters: config.Parameters{},
+					},
+				},
+			},
+			project.ConfigsPerType{
+				"test-type": []config.Config{
+					{
+						Template: template.NewInMemoryTemplate("test-tmpl", `{"key": "value\n{{ .extractedIDs.id_CLOUD_APPLICATION_0000000000000000 }}"}`),
+						Parameters: config.Parameters{
+							"extractedIDs": value.New(map[string]string{"id_CLOUD_APPLICATION_0000000000000000": "CLOUD_APPLICATION-0000000000000000"}),
+						},
+					},
+				},
+			},
+		},
 	}
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			got, gotErr := ExtractIDsIntoYAML(tt.given)
 			assert.NoError(t, gotErr)
-			assert.Equal(t, tt.want, got)
+
+			if diff := cmp.Diff(got, tt.want, cmpopts.EquateComparable(template.InMemoryTemplate{})); diff != "" {
+				t.Errorf("ExtractIDsIntoYAML() returned diff (-got +want):\n%s", diff)
+			}
 		})
 	}
 }
@@ -464,15 +490,34 @@ func TestFindAllIds(t *testing.T) {
 			"HELLO-Imnotanentityidbutstilliwasdetectedassuch",
 			nil,
 		},
+		{
+			`\nABC-0000000000000000`,
+			[]string{"ABC-0000000000000000"},
+		},
+		{
+			`\rABC-0000000000000000`,
+			[]string{"ABC-0000000000000000"},
+		},
+		{
+			`\tABC-0000000000000000`,
+			[]string{"ABC-0000000000000000"},
+		},
+		{
+			`\\tABC-0000000000000000`,
+			[]string{"ABC-0000000000000000"},
+		},
+		{
+			`\nf1614cf1-4f6e-4187-b303-af4beb42268c`,
+			[]string{"f1614cf1-4f6e-4187-b303-af4beb42268c"},
+		},
 	}
 
 	for _, tt := range tc {
-		tt := tt
 		t.Run(tt.in, func(t *testing.T) {
 			t.Parallel()
 
-			f := findAllIds(tt.in)
-			assert.ElementsMatch(t, tt.expectedIds, f)
+			foundIds := findAllIds(tt.in)
+			assert.ElementsMatch(t, tt.expectedIds, foundIds)
 		})
 	}
 


Reviews:
user: login: arthurpitman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:00:15Z
user: login: arthurpitman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:05:44Z
user: login: Laubi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:11:50Z
user: login: arthurpitman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:22:09Z
user: login: arthurpitman
body: After some consideration I'm not sure about this: the fix assumes that ME ids essentially match `[A-Z_]...}` but in that case, why not update `meIDRegexPattern` to begin with? Admittedly, this is more cautious, excluding _exactly those cases_, but it does feel pretty hacky. @tomazpu, what do you think? 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T16:47:11Z
user: login: arthurpitman
body: After quite some discussion, I feel that this is sufficient to handle exactly the case of removing a potential prefix of `n`, `r`, or `t` from a detected ME ID that begins with uppercase characters. This is a low risk option, without any significant performance penalty.
Further options, such as modifying the original ME ID detection regex, trying to remove encoded white space or unencoding the input string will be investigated in a follow up.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T09:50:40Z
user: login: tomazpu
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T11:20:54Z

Review comments:
diff_hunk: @@ -95,9 +95,23 @@ func ExtractIDsIntoYAML(configsPerType project.ConfigsPerType) (project.ConfigsP
 	return configsPerType, nil
 }
 
+// invalidMeId finds ME IDs in the form of `nABC`, `rABC`, and `tABC`. They are most commonly mistakenly created by `\nABC`.
+var invalidMeId = regexp.MustCompile("[nrt][A-Z]+")
path: pkg/download/id_extraction/id_extraction.go
user: login: arthurpitman
body: Why is this defined like this when the ME ID is a much more general `[a-zA-Z_]+-[A-Fa-f0-9]{16}`?
created_at: 2025-03-27T16:00:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,9 +95,23 @@ func ExtractIDsIntoYAML(configsPerType project.ConfigsPerType) (project.ConfigsP
 	return configsPerType, nil
 }
 
+// invalidMeId finds ME IDs in the form of `nABC`, `rABC`, and `tABC`. They are most commonly mistakenly created by `\nABC`.
+var invalidMeId = regexp.MustCompile("[nrt][A-Z]+")
path: pkg/download/id_extraction/id_extraction.go
user: login: arthurpitman
body: e.g. `nmz-1234567890ABCDEF` from `\nmz-1234567890ABCDEF` would match `meIDRegexPattern` but not `invalidMeId`.
created_at: 2025-03-27T16:05:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,9 +95,23 @@ func ExtractIDsIntoYAML(configsPerType project.ConfigsPerType) (project.ConfigsP
 	return configsPerType, nil
 }
 
+// invalidMeId finds ME IDs in the form of `nABC`, `rABC`, and `tABC`. They are most commonly mistakenly created by `\nABC`.
+var invalidMeId = regexp.MustCompile("[nrt][A-Z]+")
path: pkg/download/id_extraction/id_extraction.go
user: login: Laubi
body: TBH, I've never seen a lowercase ME ID, and I think settings even does not allow it.
The sanity check is exactly having a lowercase `n` followed by uppercase characters. If there are only lowercase characters, how can we say that it's not part of the ME ID?

Maybe we need to challenge our assumption that ME IDs can be lowercase, and that would fix it as well.
created_at: 2025-03-27T16:11:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,9 +95,23 @@ func ExtractIDsIntoYAML(configsPerType project.ConfigsPerType) (project.ConfigsP
 	return configsPerType, nil
 }
 
+// invalidMeId finds ME IDs in the form of `nABC`, `rABC`, and `tABC`. They are most commonly mistakenly created by `\nABC`.
+var invalidMeId = regexp.MustCompile("[nrt][A-Z]+")
path: pkg/download/id_extraction/id_extraction.go
user: login: arthurpitman
body: OK, that makes sense. The extra check handles exactly this case.
created_at: 2025-03-27T16:22:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-26T15:51:04Z
body: ## Unit Test Results
1 948 tests  +6   1 947 ✅ +6   53s ⏱️ ±0s
  134 suites ±0       1 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit ab48ca54. ± Comparison against base commit ded1255b.

[test-results]:data:application/gzip;base64,H4sIAGYi5GcC/12MSw7CIBQAr9KwdgEFLPUy5vFLiG0xfFaNdxehWuJyZpLZkXWLieg2kMuAYnapAWUFdQ6QnN+K4LRwaanWmYkv3mNWqrnpdA/3PJZNWHBLEfgnTAg+HCbkrbt+6G/a1Pms3C0r90fl19WlAggkEwo4w9IYQgWfMIzGXhm2DMg4a8mp1MJQ9HoD5DHMDwoBAAA=

author_association: NONE
user: login: sonarqubecloud[bot]
created_at: 2025-03-28T11:33:04Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=Dynatrace_dynatrace-configuration-as-code&pullRequest=1810)


author_association: NONE