Repository: Azure/WALinuxAgent
Pull request #3282 (2024-12-26 18:00:43)

Clean history files
Clean URLS in history files.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
Attention: Patch coverage is `68.96552%` with `18 lines` in your changes missing coverage. Please review.
> Project coverage is 72.57%. Comparing base [(`3aebcdd`)](https://app.codecov.io/gh/Azure/WALinuxAgent/commit/3aebcdd781549b55356d0682d135eec6d53b7831?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) to head [(`529265f`)](https://app.codecov.io/gh/Azure/WALinuxAgent/commit/529265ffe06a57df8883b2d1a20739a75c01c409?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).
> Report is 338 commits behind head on develop.

| [Files with missing lines](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Patch % | Lines |
|---|---|---|
| [...ol/extensions\_goal\_state\_from\_extensions\_config.py](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&filepath=azurelinuxagent%2Fcommon%2Fprotocol%2Fextensions_goal_state_from_extensions_config.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGVfZnJvbV9leHRlbnNpb25zX2NvbmZpZy5weQ==) | 81.25% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) |
| [...protocol/extensions\_goal\_state\_from\_vm\_settings.py](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&filepath=azurelinuxagent%2Fcommon%2Fprotocol%2Fextensions_goal_state_from_vm_settings.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGVfZnJvbV92bV9zZXR0aW5ncy5weQ==) | 68.42% | [4 Missing and 2 partials :warning: ](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) |
| [azurelinuxagent/common/utils/textutil.py](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&filepath=azurelinuxagent%2Fcommon%2Futils%2Ftextutil.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi91dGlscy90ZXh0dXRpbC5weQ==) | 14.28% | [6 Missing :warning: ](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) |

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           develop    #3282      +/-   ##
===========================================
+ Coverage    71.97%   72.57%   +0.59%     
===========================================
  Files          103      114      +11     
  Lines        15692    17144    +1452     
  Branches      2486     2294     -192     
===========================================
+ Hits         11295    12443    +1148     
- Misses        3881     4150     +269     
- Partials       516      551      +35     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/3282?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).


Patch:
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
index bc7b451d..c910a62b 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
@@ -22,11 +22,11 @@ from collections import defaultdict
 from azurelinuxagent.common import logger
 from azurelinuxagent.common.event import add_event, WALAEventOperation
 from azurelinuxagent.common.exception import ExtensionsConfigError
-from azurelinuxagent.common.future import ustr
+from azurelinuxagent.common.future import ustr, urlparse
 from azurelinuxagent.common.protocol.extensions_goal_state import ExtensionsGoalState, GoalStateChannel, GoalStateSource
 from azurelinuxagent.common.protocol.restapi import ExtensionSettings, Extension, VMAgentFamily, ExtensionState, InVMGoalStateMetaData
 from azurelinuxagent.common.utils.textutil import parse_doc, parse_json, findall, find, findtext, getattrib, gettext, \
-    format_exception, is_str_none_or_whitespace, is_str_empty, hasattrib
+    format_exception, is_str_none_or_whitespace, is_str_empty, hasattrib, gettextxml
 
 
 class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
@@ -38,6 +38,8 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         self._text = xml_text
         self._status_upload_blob = None
         self._status_upload_blob_type = None
+        self._status_upload_blob_xml_node = None
+        self._artifacts_profile_blob_xml_node = None
         self._required_features = []
         self._on_hold = False
         self._activity_id = None
@@ -81,13 +83,13 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         if required_features_list is not None:
             self._parse_required_features(required_features_list)
 
-        self._status_upload_blob = findtext(xml_doc, "StatusUploadBlob")
-
-        status_upload_node = find(xml_doc, "StatusUploadBlob")
-        self._status_upload_blob_type = getattrib(status_upload_node, "statusBlobType")
+        self._status_upload_blob_xml_node = find(xml_doc, "StatusUploadBlob")
+        self._status_upload_blob = gettext(self._status_upload_blob_xml_node)
+        self._status_upload_blob_type = getattrib(self._status_upload_blob_xml_node, "statusBlobType")
         logger.verbose("Extension config shows status blob type as [{0}]", self._status_upload_blob_type)
 
-        self._on_hold = ExtensionsGoalStateFromExtensionsConfig._fetch_extensions_on_hold(xml_doc, wire_client)
+        self._artifacts_profile_blob_xml_node = find(xml_doc, "InVMArtifactsProfileBlob")
+        self._on_hold = ExtensionsGoalStateFromExtensionsConfig._fetch_extensions_on_hold(self._artifacts_profile_blob_xml_node, wire_client)
 
         in_vm_gs_metadata = InVMGoalStateMetaData(find(xml_doc, "InVMGoalStateMetaData"))
         self._activity_id = self._string_to_id(in_vm_gs_metadata.activity_id)
@@ -95,7 +97,7 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         self._created_on_timestamp = self._ticks_to_utc_timestamp(in_vm_gs_metadata.created_on_ticks)
 
     @staticmethod
-    def _fetch_extensions_on_hold(xml_doc, wire_client):
+    def _fetch_extensions_on_hold(artifacts_profile_blob_xml_node, wire_client):
         def log_info(message):
             logger.info(message)
             add_event(op=WALAEventOperation.ArtifactsProfileBlob, message=message, is_success=True, log_event=False)
@@ -104,7 +106,7 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
             logger.warn(message)
             add_event(op=WALAEventOperation.ArtifactsProfileBlob, message=message, is_success=False, log_event=False)
 
-        artifacts_profile_blob = findtext(xml_doc, "InVMArtifactsProfileBlob")
+        artifacts_profile_blob = gettext(artifacts_profile_blob_xml_node)
         if is_str_none_or_whitespace(artifacts_profile_blob):
             log_info("ExtensionsConfig does not include a InVMArtifactsProfileBlob; will assume the VM is not on hold")
             return False
@@ -187,12 +189,30 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         return self._extensions
 
     def get_redacted_text(self):
-        text = self._text
-        for ext_handler in self._extensions:
-            for extension in ext_handler.settings:
-                if extension.protectedSettings is not None:
-                    text = text.replace(extension.protectedSettings, "*** REDACTED ***")
-        return text
+        def redact_url(unredacted, xml_node, name):
+            text_xml = gettextxml(xml_node)  # Note that we need to redact the raw XML text (which may contain escape sequences)
+            if text_xml is None:
+                return unredacted
+            parsed = urlparse(text_xml)
+            redacted = unredacted.replace(parsed.query, "***REDACTED***")
+            if redacted == unredacted:
+                raise Exception('Could not redact {0}'.format(name))
+            return redacted
+
+        try:
+            text = self._text
+            text = redact_url(text, self._status_upload_blob_xml_node, "StatusUploadBlob")
+            text = redact_url(text, self._artifacts_profile_blob_xml_node, "InVMArtifactsProfileBlob")
+            for ext_handler in self._extensions:
+                for extension in ext_handler.settings:
+                    if extension.protectedSettings is not None:
+                        original = text
+                        text = text.replace(extension.protectedSettings, "***REDACTED***")
+                        if text == original:
+                            return 'Could not redact protectedSettings for {0}'.format(extension.name)
+            return text
+        except Exception as e:
+            return "Error redacting text: {0}".format(e)
 
     def _parse_required_features(self, required_features_list):
         for required_feature in findall(required_features_list, "RequiredFeature"):
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py b/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
index 02f2ab59..6faeefb6 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
@@ -17,13 +17,12 @@
 # Requires Python 2.6+ and Openssl 1.0+
 import datetime
 import json
-import re
 import sys
 
 from azurelinuxagent.common import logger
 from azurelinuxagent.common.AgentGlobals import AgentGlobals
 from azurelinuxagent.common.event import WALAEventOperation, add_event
-from azurelinuxagent.common.future import ustr
+from azurelinuxagent.common.future import ustr, urlparse
 from azurelinuxagent.common.protocol.extensions_goal_state import ExtensionsGoalState, GoalStateChannel, VmSettingsParseError
 from azurelinuxagent.common.protocol.restapi import VMAgentFamily, Extension, ExtensionRequestedState, ExtensionSettings
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
@@ -143,7 +142,27 @@ class ExtensionsGoalStateFromVmSettings(ExtensionsGoalState):
         return self._extensions
 
     def get_redacted_text(self):
-        return re.sub(r'("protectedSettings"\s*:\s*)"[^"]+"', r'\1"*** REDACTED ***"', self._text)
+        try:
+            text = self._text
+
+            if self.status_upload_blob is not None:
+                parsed = urlparse(self.status_upload_blob)
+                original = text
+                text = text.replace(parsed.query, "***REDACTED***")
+                if text == original:
+                    raise Exception('Could not redact the status upload blob')
+
+            for ext_handler in self._extensions:
+                for extension in ext_handler.settings:
+                    if extension.protectedSettings is not None:
+                        original = text
+                        text = text.replace(extension.protectedSettings, "***REDACTED***")
+                        if text == original:
+                            return 'Could not redact protectedSettings for {0}'.format(extension.name)
+
+            return text
+        except Exception as e:
+            return "Error redacting text: {0}".format(e)
 
     def _parse_vm_settings(self, json_text):
         vm_settings = _CaseFoldedDict.from_dict(json.loads(json_text))
diff --git a/azurelinuxagent/common/utils/textutil.py b/azurelinuxagent/common/utils/textutil.py
index b5e15c9f..d83d5897 100644
--- a/azurelinuxagent/common/utils/textutil.py
+++ b/azurelinuxagent/common/utils/textutil.py
@@ -75,6 +75,19 @@ def gettext(node):
     return None
 
 
+def gettextxml(node):
+    """
+    Get the raw XML of a text node
+    """
+    if node is None:
+        return None
+
+    for child in node.childNodes:
+        if child.nodeType == child.TEXT_NODE:
+            return child.toxml()
+    return None
+
+
 def findtext(root, tag, namespace=None):
     """
     Get text of node by tag and namespace under Node root.


Test patch:
diff --git a/tests/common/protocol/test_goal_state.py b/tests/common/protocol/test_goal_state.py
index 7bb48cb8..a5f89587 100644
--- a/tests/common/protocol/test_goal_state.py
+++ b/tests/common/protocol/test_goal_state.py
@@ -6,10 +6,11 @@ import datetime
 import glob
 import os
 import re
+import shutil
 import time
 
 from azurelinuxagent.common import conf
-from azurelinuxagent.common.future import httpclient
+from azurelinuxagent.common.future import httpclient, urlparse
 from azurelinuxagent.common.protocol.extensions_goal_state import GoalStateSource, GoalStateChannel
 from azurelinuxagent.common.protocol.extensions_goal_state_from_extensions_config import ExtensionsGoalStateFromExtensionsConfig
 from azurelinuxagent.common.protocol.extensions_goal_state_from_vm_settings import ExtensionsGoalStateFromVmSettings
@@ -162,40 +163,82 @@ class GoalStateTestCase(AgentTestCase, HttpRequestPredicates):
             self._assert_directory_contents(
                 self._find_history_subdirectory("234-987"), ["VmSettings.json"])
 
-    def test_it_should_redact_the_protected_settings_when_saving_to_the_history_directory(self):
-        with mock_wire_protocol(wire_protocol_data.DATA_FILE_VM_SETTINGS) as protocol:
-            protocol.mock_wire_data.set_incarnation(888)
-            protocol.mock_wire_data.set_etag(888)
+    def test_it_should_redact_extensions_config(self):
+        data_file = wire_protocol_data.DATA_FILE_IN_VM_ARTIFACTS_PROFILE.copy()
+        data_file["ext_conf"] = "wire/ext_conf_redact.xml"
+        with mock_wire_protocol(data_file, detect_protocol=False) as protocol:
+            protocol.mock_wire_data.set_incarnation(888)  # set the incarnation to a known value that we can use to find the history directory
 
             goal_state = GoalState(protocol.client, save_to_history=True)
 
-            extensions_goal_state = goal_state.extensions_goal_state
-            protected_settings = []
-            for ext_handler in extensions_goal_state.extensions:
-                for extension in ext_handler.settings:
-                    if extension.protectedSettings is not None:
-                        protected_settings.append(extension.protectedSettings)
+            if goal_state.extensions_goal_state.source != GoalStateSource.Fabric:
+                raise Exception("The test goal state should be Fabric (it is {0})".format(goal_state.extensions_goal_state.source))
+
+            protected_settings = [s.protectedSettings for s in [e.settings[0] for e in goal_state.extensions_goal_state.extensions]]
             if len(protected_settings) == 0:
                 raise Exception("The test goal state does not include any protected settings")
 
-            history_directory = self._find_history_subdirectory("888-888")
-            extensions_config_file = os.path.join(history_directory, "ExtensionsConfig.xml")
-            vm_settings_file = os.path.join(history_directory, "VmSettings.json")
-            for file_name in extensions_config_file, vm_settings_file:
-                with open(file_name, "r") as stream:
-                    file_contents = stream.read()
-
-                    for settings in protected_settings:
-                        self.assertNotIn(
-                            settings,
-                            file_contents,
-                            "The protectedSettings should not have been saved to {0}".format(file_name))
-
-                    matches = re.findall(r'"protectedSettings"\s*:\s*"\*\*\* REDACTED \*\*\*"', file_contents)
-                    self.assertEqual(
-                        len(matches),
-                        len(protected_settings),
-                        "Could not find the expected number of redacted settings in {0}.\nExpected {1}.\n{2}".format(file_name, len(protected_settings), file_contents))
+            history_directory = self._find_history_subdirectory("888")
+            extensions_config = os.path.join(history_directory, "ExtensionsConfig.xml")
+            with open(extensions_config, "r") as f:
+                history_contents = f.read()
+
+            vmap_blob = re.sub(r'(?s)(.*<InVMArtifactsProfileBlob.*>)(.*)(</InVMArtifactsProfileBlob>.*)', r'\2', goal_state.extensions_goal_state._text)
+            query = urlparse(vmap_blob).query
+            redacted = vmap_blob.replace(query, "***REDACTED***")
+            self.assertNotIn(query, history_contents, "The VMAP query string was not redacted from the history")
+            self.assertNotIn(vmap_blob, history_contents, "The VMAP URL was not redacted in the history")
+            self.assertIn(redacted, history_contents, "Could not find the redacted VMAP URL in the history")
+
+            status_blob = re.sub(r'(?s)(.*<StatusUploadBlob.*>)(.*)(</StatusUploadBlob>.*)', r'\2', goal_state.extensions_goal_state._text)
+            query = urlparse(status_blob).query
+            redacted = status_blob.replace(query, "***REDACTED***")
+            self.assertNotIn(query, history_contents, "The Status query string was not redacted from the history")
+            self.assertNotIn(status_blob, history_contents, "The Status URL was not redacted in the history")
+            self.assertIn(redacted, history_contents, "Could not find the redacted Status URL in the history")
+
+            for s in protected_settings:
+                self.assertNotIn(s, history_contents, "The protected settings were not redacted from the history")
+            matches = re.findall(r'"protectedSettings"\s*:\s*"\*\*\*REDACTED\*\*\*"', history_contents)
+            self.assertEqual(len(matches), len(protected_settings),
+                "Could not find the expected number of redacted settings in {0}.\nExpected {1}.\n{2}".format(extensions_config, len(protected_settings), history_contents))
+
+    def test_it_should_redact_vm_settings(self):
+        # NOTE: vm_settings-redact_formatted.json is the same as vm_settings-redact.json, but formatted for easier reading
+        for test_file in ["hostgaplugin/vm_settings-redact.json", "hostgaplugin/vm_settings-redact_formatted.json"]:
+            data_file = wire_protocol_data.DATA_FILE_IN_VM_ARTIFACTS_PROFILE.copy()
+            data_file["vm_settings"] = test_file
+            data_file["ETag"] = "123"
+            with mock_wire_protocol(data_file, detect_protocol=False) as protocol:
+                goal_state = GoalState(protocol.client, save_to_history=True)
+
+                if goal_state.extensions_goal_state.source != GoalStateSource.FastTrack:
+                    raise Exception("The test goal state should be FastTrack (it is {0}) [test: {1}]".format(goal_state.extensions_goal_state.source, test_file))
+
+                protected_settings = [s.protectedSettings for s in [e.settings[0] for e in goal_state.extensions_goal_state.extensions]]
+                if len(protected_settings) == 0:
+                    raise Exception("The test goal state does not include any protected settings [test: {0}]".format(test_file))
+
+                history_directory = self._find_history_subdirectory("*-123")
+                vm_settings = os.path.join(history_directory, "VmSettings.json")
+                with open(vm_settings, "r") as f:
+                    history_contents = f.read()
+
+                status_blob = goal_state.extensions_goal_state.status_upload_blob
+                query = urlparse(status_blob).query
+                redacted = status_blob.replace(query, "***REDACTED***")
+                self.assertNotIn(query, history_contents, "The Status query string was not redacted from the history [test: {0}]".format(test_file))
+                self.assertNotIn(status_blob, history_contents, "The Status URL was not redacted in the history [test: {0}]".format(test_file))
+                self.assertIn(redacted, history_contents, "Could not find the redacted Status URL in the history [test: {0}]".format(test_file))
+
+                for s in protected_settings:
+                    self.assertNotIn(s, history_contents, "The protected settings were not redacted from the history [test: {0}]".format(test_file))
+
+                matches = re.findall(r'"protectedSettings"\s*:\s*"\*\*\*REDACTED\*\*\*"', history_contents)
+                self.assertEqual(len(matches), len(protected_settings),
+                    "Could not find the expected number of redacted settings in {0} [test {1}].\nExpected {2}.\n{3}".format(vm_settings, test_file, len(protected_settings), history_contents))
+
+            shutil.rmtree(history_directory)  # clean up the history directory in-between test cases to avoid stale history files
 
     def test_it_should_save_vm_settings_on_parse_errors(self):
         with mock_wire_protocol(wire_protocol_data.DATA_FILE_VM_SETTINGS) as protocol:
diff --git a/tests/data/hostgaplugin/vm_settings-redact.json b/tests/data/hostgaplugin/vm_settings-redact.json
new file mode 100644
index 00000000..5d721683
--- /dev/null
+++ b/tests/data/hostgaplugin/vm_settings-redact.json
@@ -0,0 +1,1 @@
+{"hostGAPluginVersion":"1.0.8.159","activityId":"845594b9-d46c-4649-9f3d-be3bbf795a21","correlationId":"2f027dce-6fba-4b5e-a07f-ae7d8097cc6d","inSvdSeqNo":69,"extensionsLastModifiedTickCount":638693753947160515,"extensionGoalStatesSource":"FastTrack","statusUploadBlob":{"statusBlobType":"PageBlob","value":"https://md-z9z999zzzz99.z38.blob.storage.azure.net/$system/vm01.667ae9f0-zz99-99z9-9zz9-9z9z99999zz9.status?sv=2018-03-28&sr=b&sk=system-1&sig=9ZzZZzZZZZ999zzz%2bzz99zzZzZZZ99999zz%2b9ZZZ%3d&se=9999-01-01T00%3a00%3a00Z&sp=w"},"gaFamilies":[{"name":"Prod","version":"2.12.0.2","isVersionFromRSM":false,"isVMEnabledForRSMUpgrades":false,"uris":["https://umsaq0ksgznjhnhvqvmj.blob.core.windows.net/568bb00f-455e-32b8-8deb-0e1bf1636254/568bb00f-455e-32b8-8deb-0e1bf1636254_manifest.xml"]}],"extensionGoalStates":[{"name":"Microsoft.Azure.Extensions.CustomScript","version":"2.1.10","location":"https://umsalhsl5scrdmt03dql.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml","failoverLocation":"https://umsaff4lwzr0mqltcdsj.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml","additionalLocations":["https://umsalvlwk4lfvxs4w05v.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml"],"state":"enabled","autoUpgrade":true,"runAsStartupTask":false,"isJson":true,"useExactVersion":true,"settingsSeqNo":1,"isMultiConfig":false,"settings":[{"protectedSettingsCertThumbprint":"FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4","protectedSettings":"MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.Extensions.CustomScript/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw=="}]},{"name":"Microsoft.Azure.AzureDefenderForServers.MDE.Linux","version":"1.0.4.7","location":"https://umsavph101h3qhc1pz3t.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml","failoverLocation":"https://umsazq0spn4w5z3grwcv.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml","additionalLocations":["https://umsargnfzcnssnpfskrr.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml"],"state":"enabled","autoUpgrade":true,"runAsStartupTask":false,"isJson":true,"useExactVersion":true,"settingsSeqNo":12,"isMultiConfig":false,"settings":[{"protectedSettingsCertThumbprint":"FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4","protectedSettings":"MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.AzureDefenderForServers.MDE.Linux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==","publicSettings":"{\"azureResourceId\":\"/subscriptions/z99z9999/resourceGroups/RG/providers/Microsoft.Compute/virtualMachines/vm01\",\"forceReOnboarding\":false,\"vNextEnabled\":false,\"autoUpdate\":true}"}]},{"name":"Microsoft.CPlat.Core.RunCommandLinux","version":"1.0.5","location":"https://umsac5lzx3m5mmb1qsnk.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml","failoverLocation":"https://umsarmtx2bjcqg0cj14s.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml","additionalLocations":["https://umsapqg4jwbpzxdrjjwr.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml"],"state":"enabled","autoUpgrade":true,"runAsStartupTask":false,"isJson":true,"useExactVersion":true,"settingsSeqNo":13,"isMultiConfig":false,"settings":[{"protectedSettingsCertThumbprint":"FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4","protectedSettings":"MIIC4AYJKoZIhvcNAQcWMicrosoft.CPlat.Core.RunCommandLinux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==","publicSettings":"{}"}]}]}
\ No newline at end of file
diff --git a/tests/data/hostgaplugin/vm_settings-redact_formatted.json b/tests/data/hostgaplugin/vm_settings-redact_formatted.json
new file mode 100644
index 00000000..fd5df201
--- /dev/null
+++ b/tests/data/hostgaplugin/vm_settings-redact_formatted.json
@@ -0,0 +1,93 @@
+{
+    "hostGAPluginVersion": "1.0.8.159",
+    "activityId": "845594b9-d46c-4649-9f3d-be3bbf795a21",
+    "correlationId": "2f027dce-6fba-4b5e-a07f-ae7d8097cc6d",
+    "inSvdSeqNo": 69,
+    "extensionsLastModifiedTickCount": 638693753947160515,
+    "extensionGoalStatesSource": "FastTrack",
+    "statusUploadBlob": {
+        "statusBlobType": "PageBlob",
+        "value": "https://md-z9z999zzzz99.z38.blob.storage.azure.net/$system/vm01.667ae9f0-zz99-99z9-9zz9-9z9z99999zz9.status?sv=2018-03-28&sr=b&sk=system-1&sig=9ZzZZzZZZZ999zzz%2bzz99zzZzZZZ99999zz%2b9ZZZ%3d&se=9999-01-01T00%3a00%3a00Z&sp=w"
+    },
+    "gaFamilies": [
+        {
+            "name": "Prod",
+            "version": "2.12.0.2",
+            "isVersionFromRSM": false,
+            "isVMEnabledForRSMUpgrades": false,
+            "uris": [
+                "https://umsaq0ksgznjhnhvqvmj.blob.core.windows.net/568bb00f-455e-32b8-8deb-0e1bf1636254/568bb00f-455e-32b8-8deb-0e1bf1636254_manifest.xml"
+            ]
+        }
+    ],
+    "extensionGoalStates": [
+        {
+            "name": "Microsoft.Azure.Extensions.CustomScript",
+            "version": "2.1.10",
+            "location": "https://umsalhsl5scrdmt03dql.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml",
+            "failoverLocation": "https://umsaff4lwzr0mqltcdsj.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml",
+            "additionalLocations": [
+                "https://umsalvlwk4lfvxs4w05v.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml"
+            ],
+            "state": "enabled",
+            "autoUpgrade": true,
+            "runAsStartupTask": false,
+            "isJson": true,
+            "useExactVersion": true,
+            "settingsSeqNo": 1,
+            "isMultiConfig": false,
+            "settings": [
+                {
+                    "protectedSettingsCertThumbprint": "FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4",
+                    "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.Extensions.CustomScript/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw=="
+                }
+            ]
+        },
+        {
+            "name": "Microsoft.Azure.AzureDefenderForServers.MDE.Linux",
+            "version": "1.0.4.7",
+            "location": "https://umsavph101h3qhc1pz3t.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml",
+            "failoverLocation": "https://umsazq0spn4w5z3grwcv.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml",
+            "additionalLocations": [
+                "https://umsargnfzcnssnpfskrr.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml"
+            ],
+            "state": "enabled",
+            "autoUpgrade": true,
+            "runAsStartupTask": false,
+            "isJson": true,
+            "useExactVersion": true,
+            "settingsSeqNo": 12,
+            "isMultiConfig": false,
+            "settings": [
+                {
+                    "protectedSettingsCertThumbprint": "FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4",
+                    "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.AzureDefenderForServers.MDE.Linux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==",
+                    "publicSettings": "{\"azureResourceId\":\"/subscriptions/z99z9999/resourceGroups/RG/providers/Microsoft.Compute/virtualMachines/vm01\",\"forceReOnboarding\":false,\"vNextEnabled\":false,\"autoUpdate\":true}"
+                }
+            ]
+        },
+        {
+            "name": "Microsoft.CPlat.Core.RunCommandLinux",
+            "version": "1.0.5",
+            "location": "https://umsac5lzx3m5mmb1qsnk.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml",
+            "failoverLocation": "https://umsarmtx2bjcqg0cj14s.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml",
+            "additionalLocations": [
+                "https://umsapqg4jwbpzxdrjjwr.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml"
+            ],
+            "state": "enabled",
+            "autoUpgrade": true,
+            "runAsStartupTask": false,
+            "isJson": true,
+            "useExactVersion": true,
+            "settingsSeqNo": 13,
+            "isMultiConfig": false,
+            "settings": [
+                {
+                    "protectedSettingsCertThumbprint": "FF56DAC2F36EDDE292DE9D49B200D7CBEE05D2F4",
+                    "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.CPlat.Core.RunCommandLinux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==",
+                    "publicSettings": "{}"
+                }
+            ]
+        }
+    ]
+}
\ No newline at end of file
diff --git a/tests/data/wire/ext_conf_redact.xml b/tests/data/wire/ext_conf_redact.xml
new file mode 100644
index 00000000..b75b2ed3
--- /dev/null
+++ b/tests/data/wire/ext_conf_redact.xml
@@ -0,0 +1,76 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Extensions version="1.0.0.0" goalStateIncarnation="1"><GuestAgentExtension xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
+  <GAFamilies>
+    <GAFamily>
+      <IsVMEnabledForRSMUpgrades>false</IsVMEnabledForRSMUpgrades>
+      <IsVersionFromRSM>false</IsVersionFromRSM>
+      <Name>Prod</Name>
+      <Uris>
+        <Uri>https://umsaq0ksgznjhnhvqvmj.blob.core.windows.net/568bb00f-455e-32b8-8deb-0e1bf1636254/568bb00f-455e-32b8-8deb-0e1bf1636254_manifest.xml</Uri>
+      </Uris>
+      <Version>2.12.0.2</Version>
+    </GAFamily>
+  </GAFamilies>
+  <Location>westus2</Location>
+  <ServiceName>CRP</ServiceName>
+</GuestAgentExtension>
+<StatusUploadBlob statusBlobType="PageBlob">https://md-z9z999zzzz99.z38.blob.storage.azure.net/$system/vm01.667ae9f0-zz99-99z9-9zz9-9z9z99999zz9.status?sv=2018-03-28&amp;sr=b&amp;sk=system-1&amp;sig=9ZzZZzZZZZ999zzz%2bzz99zzZzZZZ99999zz%2b9ZZZ%3d&amp;se=9999-01-01T00%3a00%3a00Z&amp;sp=w</StatusUploadBlob>
+<Plugins>
+  <Plugin name="Microsoft.Azure.Extensions.CustomScript" version="2.1.10" location="https://umsaff4lwzr0mqltcdsj.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml" state="enabled" autoUpgrade="true" failoverlocation="https://umsat3kgcjtz4rzmtdhx.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml" runAsStartupTask="false" isJson="true" useExactVersion="true">
+    <additionalLocations>
+      <additionalLocation>https://umsalhsl5scrdmt03dql.blob.core.windows.net/5237dd14-0aad-f051-0fad-1e33e1b63091/5237dd14-0aad-f051-0fad-1e33e1b63091_manifest.xml</additionalLocation>
+    </additionalLocations>
+  </Plugin>
+  <Plugin name="Microsoft.Azure.AzureDefenderForServers.MDE.Linux" version="1.0.4.7" location="https://umsargnfzcnssnpfskrr.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml" state="enabled" autoUpgrade="true" failoverlocation="https://umsajn4ck0sd50j02kxr.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml" runAsStartupTask="false" isJson="true" useExactVersion="true">
+    <additionalLocations>
+      <additionalLocation>https://umsajcsg1mkf30sqkz4v.blob.core.windows.net/60eb5f5d-256d-1d4b-3db7-8808756d9b09/60eb5f5d-256d-1d4b-3db7-8808756d9b09_manifest.xml</additionalLocation>
+    </additionalLocations>
+  </Plugin>
+  <Plugin name="Microsoft.CPlat.Core.RunCommandLinux" version="1.0.5" location="https://umsapqg4jwbpzxdrjjwr.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml" state="enabled" autoUpgrade="true" failoverlocation="https://umsac5lzx3m5mmb1qsnk.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml" runAsStartupTask="false" isJson="true" useExactVersion="true">
+    <additionalLocations>
+      <additionalLocation>https://umsarmtx2bjcqg0cj14s.blob.core.windows.net/4beefb67-5a1e-aa37-7ede-3180ba34ba0d/4beefb67-5a1e-aa37-7ede-3180ba34ba0d_manifest.xml</additionalLocation>
+    </additionalLocations>
+  </Plugin>
+</Plugins>
+<PluginSettings>
+  <Plugin name="Microsoft.Azure.Extensions.CustomScript" version="2.1.10">
+    <RuntimeSettings seqNo="1">{
+  "runtimeSettings": [
+    {
+      "handlerSettings": {
+        "protectedSettingsCertThumbprint": "3CA9F246CB9F2D4D17A1BBF732CA5D70AE5CB810",
+        "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.Extensions.CustomScript/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw=="
+      }
+    }
+  ]
+}</RuntimeSettings>
+  </Plugin>
+  <Plugin name="Microsoft.Azure.AzureDefenderForServers.MDE.Linux" version="1.0.4.7">
+    <RuntimeSettings seqNo="12">{
+  "runtimeSettings": [
+    {
+      "handlerSettings": {
+        "protectedSettingsCertThumbprint": "3CA9F246CB9F2D4D17A1BBF732CA5D70AE5CB810",
+        "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.Azure.AzureDefenderForServers.MDE.Linux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==",
+        "publicSettings": {"azureResourceId":"/subscriptions/z99z9999/resourceGroups/RG/providers/Microsoft.Compute/virtualMachines/vm01","forceReOnboarding":false,"vNextEnabled":false,"autoUpdate":true}
+      }
+    }
+  ]
+}</RuntimeSettings>
+  </Plugin>
+  <Plugin name="Microsoft.CPlat.Core.RunCommandLinux" version="1.0.5">
+    <RuntimeSettings seqNo="13">{
+  "runtimeSettings": [
+    {
+      "handlerSettings": {
+        "protectedSettingsCertThumbprint": "3CA9F246CB9F2D4D17A1BBF732CA5D70AE5CB810",
+        "protectedSettings": "MIIC4AYJKoZIhvcNAQcWMicrosoft.CPlat.Core.RunCommandLinux/IsZAEZFidXVFB/8Ucc8OxGFWpqtXZdrFcuuSxOa6ib/+la5ukd7pXjzYCd/Sp9yGj7Rw3Kqf9Lpix/sPuqWjV6e2XgDhW4j2nRNTJU4yXxocQm/PhAr39Um7n0pgI2Cn28AabYtsHWjKqr8Al9LX6bKm8cnmnLjqTntphCw==",
+        "publicSettings": {}
+      }
+    }
+  ]
+}</RuntimeSettings>
+  </Plugin>
+</PluginSettings>
+<InVMArtifactsProfileBlob>https://md-z9z999zzzz99.z38.blob.storage.azure.net/$system/vm01.667ae9f0-zz99-99z9-9zz9-9z9z99999zz9.vmSettings?sv=2018-03-28&amp;sr=b&amp;sk=system-1&amp;sig=9ZzZZZ%2fZzz9zZzZZZZz9z9zZ99Zzz%3d&amp;se=9999-01-01T00%3a00%3a00Z&amp;sp=r</InVMArtifactsProfileBlob>
+<InVMGoalStateMetaData inSvdSeqNo="71" createdOnTicks="638697286354582862" activityId="b72a79fc-6f88-41f5-9f9c-0f3b808de282" correlationId="ac50d27a-b3db-411f-a74e-aa8c0e358258" /></Extensions>
\ No newline at end of file