Repository: Azure/WALinuxAgent
Pull request #2655 (2022-08-19 23:24:07)

Add x-ms-verify-from-artifacts-blob to extensionsArtifact calls
The HostGAPlugin requires the x-ms-verify-from-artifacts-blob header in extensionsArtifact requests for Fast Track Goal states.

This PR adds that header. Most of the changes in the PR are code refactoring that makes this simple, plus some refactoring that has been delayed in previous changes, as well as a few changes for Python 3.10

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
> Merging [#2655](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (8b13620) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/3aebcdd781549b55356d0682d135eec6d53b7831?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (3aebcdd) will **decrease** coverage by `0.00%`.
> The diff coverage is `94.16%`.

```diff
@@             Coverage Diff             @@
##           develop    #2655      +/-   ##
===========================================
- Coverage    71.97%   71.97%   -0.01%     
===========================================
  Files          103      103              
  Lines        15692    15683       -9     
  Branches      2486     2487       +1     
===========================================
- Hits         11295    11288       -7     
+ Misses        3881     3879       -2     
  Partials       516      516              
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/protocol/hostplugin.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9ob3N0cGx1Z2luLnB5) | `87.23% <66.66%> (-0.44%)` | :arrow_down: |
| [azurelinuxagent/common/protocol/goal\_state.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9nb2FsX3N0YXRlLnB5) | `94.58% <89.36%> (-0.74%)` | :arrow_down: |
| [...inuxagent/common/protocol/extensions\_goal\_state.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGUucHk=) | `74.46% <100.00%> (ø)` | |
| [...ol/extensions\_goal\_state\_from\_extensions\_config.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGVfZnJvbV9leHRlbnNpb25zX2NvbmZpZy5weQ==) | `92.05% <100.00%> (ø)` | |
| [...protocol/extensions\_goal\_state\_from\_vm\_settings.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGVfZnJvbV92bV9zZXR0aW5ncy5weQ==) | `83.45% <100.00%> (ø)` | |
| [azurelinuxagent/common/protocol/restapi.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9yZXN0YXBpLnB5) | `97.50% <100.00%> (ø)` | |
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `77.83% <100.00%> (-0.71%)` | :arrow_down: |
| [azurelinuxagent/common/utils/archive.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi91dGlscy9hcmNoaXZlLnB5) | `81.92% <100.00%> (+0.20%)` | :arrow_up: |
| [azurelinuxagent/ga/exthandlers.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL2V4dGhhbmRsZXJzLnB5) | `85.70% <100.00%> (+0.01%)` | :arrow_up: |
| [azurelinuxagent/ga/update.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL3VwZGF0ZS5weQ==) | `89.43% <100.00%> (+0.09%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/Azure/WALinuxAgent/pull/2655/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)


Patch:
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state.py b/azurelinuxagent/common/protocol/extensions_goal_state.py
index 74c761ca..337cff11 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state.py
@@ -17,7 +17,7 @@
 # Requires Python 2.6+ and Openssl 1.0+
 import datetime
 
-import azurelinuxagent.common.logger as logger
+from azurelinuxagent.common import logger
 from azurelinuxagent.common.AgentGlobals import AgentGlobals
 from azurelinuxagent.common.exception import AgentError
 from azurelinuxagent.common.utils import textutil
@@ -126,7 +126,7 @@ class ExtensionsGoalState(object):
         raise NotImplementedError()
 
     @property
-    def agent_manifests(self):
+    def agent_families(self):
         raise NotImplementedError()
 
     @property
@@ -233,7 +233,7 @@ class EmptyExtensionsGoalState(ExtensionsGoalState):
         return False
 
     @property
-    def agent_manifests(self):
+    def agent_families(self):
         return []
 
     @property
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
index 27af3b79..a8bfa250 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
@@ -24,7 +24,7 @@ from azurelinuxagent.common.event import add_event, WALAEventOperation
 from azurelinuxagent.common.exception import ExtensionsConfigError
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.extensions_goal_state import ExtensionsGoalState, GoalStateChannel, GoalStateSource
-from azurelinuxagent.common.protocol.restapi import ExtensionSettings, Extension, VMAgentManifest, ExtensionState, InVMGoalStateMetaData
+from azurelinuxagent.common.protocol.restapi import ExtensionSettings, Extension, VMAgentFamily, ExtensionState, InVMGoalStateMetaData
 from azurelinuxagent.common.utils.textutil import parse_doc, parse_json, findall, find, findtext, getattrib, gettext, format_exception, \
     is_str_none_or_whitespace, is_str_empty
 
@@ -43,7 +43,7 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         self._activity_id = None
         self._correlation_id = None
         self._created_on_timestamp = None
-        self._agent_manifests = []
+        self._agent_families = []
         self._extensions = []
 
         try:
@@ -59,14 +59,14 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         ga_families = findall(ga_families_list, "GAFamily")
 
         for ga_family in ga_families:
-            family = findtext(ga_family, "Name")
+            name = findtext(ga_family, "Name")
             version = findtext(ga_family, "Version")
             uris_list = find(ga_family, "Uris")
             uris = findall(uris_list, "Uri")
-            manifest = VMAgentManifest(family, version)
+            family = VMAgentFamily(name, version)
             for uri in uris:
-                manifest.uris.append(gettext(uri))
-            self._agent_manifests.append(manifest)
+                family.uris.append(gettext(uri))
+            self._agent_families.append(family)
 
         self.__parse_plugins_and_settings_and_populate_ext_handlers(xml_doc)
 
@@ -172,8 +172,8 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
         return self._on_hold
 
     @property
-    def agent_manifests(self):
-        return self._agent_manifests
+    def agent_families(self):
+        return self._agent_families
 
     @property
     def extensions(self):
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py b/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
index db7237b9..f6496bfd 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state_from_vm_settings.py
@@ -20,11 +20,11 @@ import json
 import re
 import sys
 
+from azurelinuxagent.common import logger
 from azurelinuxagent.common.AgentGlobals import AgentGlobals
 from azurelinuxagent.common.future import ustr
-import azurelinuxagent.common.logger as logger
 from azurelinuxagent.common.protocol.extensions_goal_state import ExtensionsGoalState, GoalStateChannel, VmSettingsParseError
-from azurelinuxagent.common.protocol.restapi import VMAgentManifest, Extension, ExtensionRequestedState, ExtensionSettings
+from azurelinuxagent.common.protocol.restapi import VMAgentFamily, Extension, ExtensionRequestedState, ExtensionSettings
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 
 
@@ -48,7 +48,7 @@ class ExtensionsGoalStateFromVmSettings(ExtensionsGoalState):
         self._status_upload_blob_type = None
         self._required_features = []
         self._on_hold = False
-        self._agent_manifests = []
+        self._agent_families = []
         self._extensions = []
 
         try:
@@ -134,8 +134,8 @@ class ExtensionsGoalStateFromVmSettings(ExtensionsGoalState):
         return self._on_hold
 
     @property
-    def agent_manifests(self):
-        return self._agent_manifests
+    def agent_families(self):
+        return self._agent_families
 
     @property
     def extensions(self):
@@ -269,10 +269,10 @@ class ExtensionsGoalStateFromVmSettings(ExtensionsGoalState):
             uris = family.get("uris")
             if uris is None:
                 uris = []
-            manifest = VMAgentManifest(name, version)
+            agent_family = VMAgentFamily(name, version)
             for u in uris:
-                manifest.uris.append(u)
-            self._agent_manifests.append(manifest)
+                agent_family.uris.append(u)
+            self._agent_families.append(agent_family)
 
     def _parse_extensions(self, vm_settings):
         # Sample (NOTE: The first sample is single-config, the second multi-config):
diff --git a/azurelinuxagent/common/protocol/goal_state.py b/azurelinuxagent/common/protocol/goal_state.py
index 97ae270f..c2b95cfb 100644
--- a/azurelinuxagent/common/protocol/goal_state.py
+++ b/azurelinuxagent/common/protocol/goal_state.py
@@ -21,8 +21,8 @@ import re
 import time
 import json
 
-import azurelinuxagent.common.conf as conf
-import azurelinuxagent.common.logger as logger
+from azurelinuxagent.common import conf
+from azurelinuxagent.common import logger
 from azurelinuxagent.common.AgentGlobals import AgentGlobals
 from azurelinuxagent.common.datacontract import set_properties
 from azurelinuxagent.common.event import add_event, WALAEventOperation
@@ -31,11 +31,11 @@ from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.extensions_goal_state_factory import ExtensionsGoalStateFactory
 from azurelinuxagent.common.protocol.extensions_goal_state import VmSettingsParseError, GoalStateSource
 from azurelinuxagent.common.protocol.hostplugin import VmSettingsNotSupported, VmSettingsSupportStopped
-from azurelinuxagent.common.protocol.restapi import Cert, CertList, RemoteAccessUser, RemoteAccessUsersList
+from azurelinuxagent.common.protocol.restapi import Cert, CertList, RemoteAccessUser, RemoteAccessUsersList, ExtHandlerPackage, ExtHandlerPackageList
 from azurelinuxagent.common.utils import fileutil
 from azurelinuxagent.common.utils.archive import GoalStateHistory, SHARED_CONF_FILE_NAME
 from azurelinuxagent.common.utils.cryptutil import CryptUtil
-from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, findtext, getattrib
+from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, findtext, getattrib, gettext
 
 
 GOAL_STATE_URI = "http://{0}/machine/?comp=goalstate"
@@ -129,6 +129,36 @@ class GoalState(object):
     def remote_access(self):
         return self._remote_access
 
+    def fetch_agent_manifest(self, family_name, uris):
+        """
+        This is a convenience method that wraps WireClient.fetch_manifest(), but adds the required 'use_verify_header' parameter and saves
+        the manifest to the history folder.
+        """
+        return self._fetch_manifest("agent", "waagent.{0}".format(family_name), uris)
+
+    def fetch_extension_manifest(self, extension_name, uris):
+        """
+        This is a convenience method that wraps WireClient.fetch_manifest(), but adds the required 'use_verify_header' parameter and saves
+        the manifest to the history folder.
+        """
+        return self._fetch_manifest("extension", extension_name, uris)
+
+    def _fetch_manifest(self, manifest_type, name, uris):
+        try:
+            is_fast_track = self.extensions_goal_state.source == GoalStateSource.FastTrack
+            xml_text = self._wire_client.fetch_manifest(uris, use_verify_header=is_fast_track)
+            self._history.save_manifest(name, xml_text)
+            return ExtensionManifest(xml_text)
+        except Exception as e:
+            raise ProtocolError("Failed to retrieve {0} manifest. Error: {1}".format(manifest_type, ustr(e)))
+
+    def download_extension(self, uris, destination, on_downloaded=lambda: True):
+        """
+        This is a convenience method that wraps WireClient.download_extension(), but adds the required 'use_verify_header' parameter.
+        """
+        is_fast_track = self.extensions_goal_state.source == GoalStateSource.FastTrack
+        self._wire_client.download_extension(uris, destination, use_verify_header=is_fast_track, on_downloaded=on_downloaded)
+
     @staticmethod
     def update_host_plugin_headers(wire_client):
         """
@@ -562,3 +592,46 @@ class RemoteAccess(object):
         remote_access_user = RemoteAccessUser(name, encrypted_password, expiration)
         return remote_access_user
 
+
+class ExtensionManifest(object):
+    def __init__(self, xml_text):
+        if xml_text is None:
+            raise ValueError("ExtensionManifest is None")
+        logger.verbose("Load ExtensionManifest.xml")
+        self.pkg_list = ExtHandlerPackageList()
+        self._parse(xml_text)
+
+    def _parse(self, xml_text):
+        xml_doc = parse_doc(xml_text)
+        self._handle_packages(findall(find(xml_doc,
+                                           "Plugins"),
+                                      "Plugin"),
+                              False)
+        self._handle_packages(findall(find(xml_doc,
+                                           "InternalPlugins"),
+                                      "Plugin"),
+                              True)
+
+    def _handle_packages(self, packages, isinternal):
+        for package in packages:
+            version = findtext(package, "Version")
+
+            disallow_major_upgrade = findtext(package,
+                                              "DisallowMajorVersionUpgrade")
+            if disallow_major_upgrade is None:
+                disallow_major_upgrade = ''
+            disallow_major_upgrade = disallow_major_upgrade.lower() == "true"
+
+            uris = find(package, "Uris")
+            uri_list = findall(uris, "Uri")
+            uri_list = [gettext(x) for x in uri_list]
+            pkg = ExtHandlerPackage()
+            pkg.version = version
+            pkg.disallow_major_upgrade = disallow_major_upgrade
+            for uri in uri_list:
+                pkg.uris.append(uri)
+
+            pkg.isinternal = isinternal
+            self.pkg_list.versions.append(pkg)
+
+
diff --git a/azurelinuxagent/common/protocol/hostplugin.py b/azurelinuxagent/common/protocol/hostplugin.py
index 583b95ec..56cbafc1 100644
--- a/azurelinuxagent/common/protocol/hostplugin.py
+++ b/azurelinuxagent/common/protocol/hostplugin.py
@@ -55,6 +55,7 @@ _HEADER_VERSION = "x-ms-version"
 _HEADER_HOST_CONFIG_NAME = "x-ms-host-config-name"
 _HEADER_ARTIFACT_LOCATION = "x-ms-artifact-location"
 _HEADER_ARTIFACT_MANIFEST_LOCATION = "x-ms-artifact-manifest-location"
+_HEADER_VERIFY_FROM_ARTIFACTS_BLOB = "x-ms-verify-from-artifacts-blob"
 
 MAXIMUM_PAGEBLOB_PAGE_SIZE = 4 * 1024 * 1024  # Max page size: 4MB
 
@@ -183,19 +184,21 @@ class HostPluginProtocol(object):
 
         return url, headers
 
-    def get_artifact_request(self, artifact_url, artifact_manifest_url=None):
+    def get_artifact_request(self, artifact_url, use_verify_header, artifact_manifest_url=None):
         if not self.ensure_initialized():
             raise ProtocolError("HostGAPlugin: Host plugin channel is not available")
 
         if textutil.is_str_none_or_whitespace(artifact_url):
             raise ProtocolError("HostGAPlugin: No extension artifact url was provided")
 
-        url = URI_FORMAT_GET_EXTENSION_ARTIFACT.format(self.endpoint,
-                                                       HOST_PLUGIN_PORT)
-        headers = {_HEADER_VERSION: API_VERSION,
-                   _HEADER_CONTAINER_ID: self.container_id,
-                   _HEADER_HOST_CONFIG_NAME: self.role_config_name,
-                   _HEADER_ARTIFACT_LOCATION: artifact_url}
+        url = URI_FORMAT_GET_EXTENSION_ARTIFACT.format(self.endpoint, HOST_PLUGIN_PORT)
+        headers = {
+            _HEADER_VERSION: API_VERSION,
+               _HEADER_CONTAINER_ID: self.container_id,
+               _HEADER_HOST_CONFIG_NAME: self.role_config_name,
+               _HEADER_ARTIFACT_LOCATION: artifact_url}
+        if use_verify_header:
+            headers[_HEADER_VERIFY_FROM_ARTIFACTS_BLOB] = "true"
 
         if artifact_manifest_url is not None:
             headers[_HEADER_ARTIFACT_MANIFEST_LOCATION] = artifact_manifest_url
diff --git a/azurelinuxagent/common/protocol/restapi.py b/azurelinuxagent/common/protocol/restapi.py
index 0edd4623..725e2d7b 100644
--- a/azurelinuxagent/common/protocol/restapi.py
+++ b/azurelinuxagent/common/protocol/restapi.py
@@ -68,9 +68,9 @@ class CertList(DataContract):
         self.certificates = DataContractList(Cert)
 
 
-class VMAgentManifest(object):
-    def __init__(self, family, version=None):
-        self.family = family
+class VMAgentFamily(object):
+    def __init__(self, name, version=None):
+        self.name = name
         # This is the Requested version as specified by the Goal State, it defaults to 0.0.0.0 if not specified in GS
         self.requested_version_string = VERSION_0 if version is None else version
         self.uris = []
@@ -91,7 +91,7 @@ class VMAgentManifest(object):
         return self.__str__()
 
     def __str__(self):
-        return "[family: '{0}' uris: {1}]".format(self.family, self.uris)
+        return "[name: '{0}' uris: {1}]".format(self.name, self.uris)
 
 
 class ExtensionState(object):
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index 6d48474d..167d4820 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -37,11 +37,9 @@ from azurelinuxagent.common.exception import ProtocolNotFoundError, \
 from azurelinuxagent.common.future import httpclient, bytebuffer, ustr
 from azurelinuxagent.common.protocol.goal_state import GoalState, TRANSPORT_CERT_FILE_NAME, TRANSPORT_PRV_FILE_NAME
 from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol
-from azurelinuxagent.common.protocol.restapi import DataContract, ExtHandlerPackage, \
-    ExtHandlerPackageList, ProvisionStatus, VMInfo, VMStatus
+from azurelinuxagent.common.protocol.restapi import DataContract, ProvisionStatus, VMInfo, VMStatus
 from azurelinuxagent.common.telemetryevent import GuestAgentExtensionEventsSchema
 from azurelinuxagent.common.utils import fileutil, restutil
-from azurelinuxagent.common.utils.archive import _MANIFEST_FILE_NAME
 from azurelinuxagent.common.utils.cryptutil import CryptUtil
 from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, \
     findtext, gettext, remove_bom, get_bytes_from_pem, parse_json
@@ -111,22 +109,6 @@ class WireProtocol(DataContract):
         certificates = self.client.get_certs()
         return certificates.cert_list
 
-    def get_vmagent_manifests(self):
-        goal_state = self.client.get_goal_state()
-        ext_conf = goal_state.extensions_goal_state
-        return ext_conf.agent_manifests, goal_state.incarnation
-
-    def get_vmagent_pkgs(self, vmagent_manifest):
-        goal_state = self.client.get_goal_state()
-        ga_manifest = self.client.fetch_gafamily_manifest(vmagent_manifest, goal_state)
-        valid_pkg_list = ga_manifest.pkg_list
-        return valid_pkg_list
-
-    def get_ext_handler_pkgs(self, ext_handler):
-        logger.verbose("Get extension handler package")
-        man = self.client.get_ext_manifest(ext_handler)
-        return man.pkg_list
-
     def get_goal_state(self):
         return self.client.get_goal_state()
 
@@ -597,14 +579,14 @@ class WireClient(object):
         return http_req(*args, **kwargs)
 
     def fetch_artifacts_profile_blob(self, uri):
-        return self._fetch_content("artifacts profile blob", [uri])[1]  # _fetch_content returns a (uri, content) tuple
+        return self._fetch_content("artifacts profile blob", [uri], use_verify_header=False)[1]  # _fetch_content returns a (uri, content) tuple
 
-    def fetch_manifest(self, uris):
-        uri, content = self._fetch_content("manifest", uris)
+    def fetch_manifest(self, uris, use_verify_header):
+        uri, content = self._fetch_content("manifest", uris, use_verify_header=use_verify_header)
         self.get_host_plugin().update_manifest_uri(uri)
         return content
 
-    def _fetch_content(self, download_type, uris):
+    def _fetch_content(self, download_type, uris, use_verify_header):
         """
         Walks the given list of 'uris' issuing HTTP GET requests; returns a tuple with the URI and the content of the first successful request.
 
@@ -616,13 +598,13 @@ class WireClient(object):
         direct_download = lambda uri: self.fetch(uri)[0]
 
         def hgap_download(uri):
-            request_uri, request_headers = host_ga_plugin.get_artifact_request(uri)
+            request_uri, request_headers = host_ga_plugin.get_artifact_request(uri, use_verify_header=use_verify_header)
             response, _ = self.fetch(request_uri, request_headers, use_proxy=False, retry_codes=restutil.HGAP_GET_EXTENSION_ARTIFACT_RETRY_CODES)
             return response
 
         return self._download_with_fallback_channel(download_type, uris, direct_download=direct_download, hgap_download=hgap_download)
 
-    def download_extension(self, uris, destination, on_downloaded=lambda: True):
+    def download_extension(self, uris, destination, use_verify_header, on_downloaded=lambda: True):
         """
         Walks the given list of 'uris' issuing HTTP GET requests and saves the content of the first successful request to 'destination'.
 
@@ -635,7 +617,7 @@ class WireClient(object):
         direct_download = lambda uri: self.stream(uri, destination, headers=None, use_proxy=True)
 
         def hgap_download(uri):
-            request_uri, request_headers = host_ga_plugin.get_artifact_request(uri, host_ga_plugin.manifest_uri)
+            request_uri, request_headers = host_ga_plugin.get_artifact_request(uri, use_verify_header=use_verify_header, artifact_manifest_url=host_ga_plugin.manifest_uri)
             return self.stream(request_uri, destination, headers=request_headers, use_proxy=False)
 
         self._download_with_fallback_channel("extension package", uris, direct_download=direct_download, hgap_download=hgap_download, on_downloaded=on_downloaded)
@@ -808,30 +790,11 @@ class WireClient(object):
             raise ProtocolError("Trying to fetch Certificates before initialization!")
         return self._goal_state.certs
 
-    def get_ext_manifest(self, ext_handler):
-        if self._goal_state is None:
-            raise ProtocolError("Trying to fetch Extension Manifest before initialization!")
-
-        try:
-            xml_text = self.fetch_manifest(ext_handler.manifest_uris)
-            self._goal_state.save_to_history(xml_text, _MANIFEST_FILE_NAME.format(ext_handler.name))
-            return ExtensionManifest(xml_text)
-        except Exception as e:
-            raise ExtensionDownloadError("Failed to retrieve extension manifest. Error: {0}".format(ustr(e)))
-
     def get_remote_access(self):
         if self._goal_state is None:
             raise ProtocolError("Trying to fetch Remote Access before initialization!")
         return self._goal_state.remote_access
 
-    def fetch_gafamily_manifest(self, vmagent_manifest, goal_state):
-        try:
-            xml_text = self.fetch_manifest(vmagent_manifest.uris)
-            goal_state.save_to_history(xml_text, _MANIFEST_FILE_NAME.format(vmagent_manifest.family))
-            return ExtensionManifest(xml_text)
-        except Exception as e:
-            raise ProtocolError("Failed to retrieve GAFamily manifest. Error: {0}".format(ustr(e)))
-
     def check_wire_protocol_version(self):
         uri = VERSION_INFO_URI.format(self.get_endpoint())
         version_info_xml = self.fetch_config(uri, None)
@@ -1186,48 +1149,6 @@ class VersionInfo(object):
         return self.supported
 
 
-class ExtensionManifest(object): 
-    def __init__(self, xml_text):
-        if xml_text is None:
-            raise ValueError("ExtensionManifest is None")
-        logger.verbose("Load ExtensionManifest.xml")
-        self.pkg_list = ExtHandlerPackageList()
-        self._parse(xml_text)
-
-    def _parse(self, xml_text):
-        xml_doc = parse_doc(xml_text)
-        self._handle_packages(findall(find(xml_doc,
-                                           "Plugins"),
-                                      "Plugin"),
-                              False)
-        self._handle_packages(findall(find(xml_doc,
-                                           "InternalPlugins"),
-                                      "Plugin"),
-                              True)
-
-    def _handle_packages(self, packages, isinternal):
-        for package in packages:
-            version = findtext(package, "Version")
-
-            disallow_major_upgrade = findtext(package,
-                                              "DisallowMajorVersionUpgrade")
-            if disallow_major_upgrade is None:
-                disallow_major_upgrade = ''
-            disallow_major_upgrade = disallow_major_upgrade.lower() == "true"
-
-            uris = find(package, "Uris")
-            uri_list = findall(uris, "Uri")
-            uri_list = [gettext(x) for x in uri_list]
-            pkg = ExtHandlerPackage()
-            pkg.version = version
-            pkg.disallow_major_upgrade = disallow_major_upgrade
-            for uri in uri_list:
-                pkg.uris.append(uri)
-
-            pkg.isinternal = isinternal
-            self.pkg_list.versions.append(pkg)
-
-
 # Do not extend this class
 class InVMArtifactsProfile(object):
     """
diff --git a/azurelinuxagent/common/utils/archive.py b/azurelinuxagent/common/utils/archive.py
index 1f8fdd93..0385b6db 100644
--- a/azurelinuxagent/common/utils/archive.py
+++ b/azurelinuxagent/common/utils/archive.py
@@ -7,8 +7,8 @@ import re
 import shutil
 import zipfile
 
-import azurelinuxagent.common.logger as logger
-import azurelinuxagent.common.conf as conf
+from azurelinuxagent.common import conf
+from azurelinuxagent.common import logger
 from azurelinuxagent.common.utils import fileutil, timeutil
 
 # pylint: disable=W0105
@@ -306,3 +306,6 @@ class GoalStateHistory(object):
 
     def save_shared_conf(self, text):
         self.save(text, SHARED_CONF_FILE_NAME)
+
+    def save_manifest(self, name, text):
+        self.save(text, _MANIFEST_FILE_NAME.format(name))
diff --git a/azurelinuxagent/ga/exthandlers.py b/azurelinuxagent/ga/exthandlers.py
index d6b5ab7b..99f38094 100644
--- a/azurelinuxagent/ga/exthandlers.py
+++ b/azurelinuxagent/ga/exthandlers.py
@@ -949,9 +949,9 @@ class ExtHandlersHandler(object):
         if status_blob_text is None:
             status_blob_text = ""
 
-        support_multi_config = dict()
+        support_multi_config = {}
         vm_status_data = get_properties(vm_status)
-        vm_handler_statuses = vm_status_data.get('vmAgent', dict()).get('extensionHandlers')
+        vm_handler_statuses = vm_status_data.get('vmAgent', {}).get('extensionHandlers')
         for handler_status in vm_handler_statuses:
             if handler_status.get('name') is not None:
                 support_multi_config[handler_status.get('name')] = handler_status.get('supports_multi_config')
@@ -1105,7 +1105,8 @@ class ExtHandlerInstance(object):
     def decide_version(self, target_state=None, extension=None):
         self.logger.verbose("Decide which version to use")
         try:
-            pkg_list = self.protocol.get_ext_handler_pkgs(self.ext_handler)
+            manifest = self.protocol.get_goal_state().fetch_extension_manifest(self.ext_handler.name, self.ext_handler.manifest_uris)
+            pkg_list = manifest.pkg_list
         except ProtocolError as e:
             raise ExtensionError("Failed to get ext handler pkgs", e)
         except ExtensionDownloadError:
@@ -1259,7 +1260,7 @@ class ExtHandlerInstance(object):
                 self.logger.info("The existing extension package is invalid, will ignore it.")
 
         if not package_exists:
-            self.protocol.client.download_extension(self.pkg.uris, destination, on_downloaded=lambda: self._unzip_extension_package(destination, self.get_base_dir()))
+            self.protocol.get_goal_state().download_extension(self.pkg.uris, destination, on_downloaded=lambda: self._unzip_extension_package(destination, self.get_base_dir()))
             self.report_event(message="Download succeeded", duration=elapsed_milliseconds(begin_utc))
 
         self.pkg_file = destination
diff --git a/azurelinuxagent/ga/update.py b/azurelinuxagent/ga/update.py
index 68c0fb4e..0fac66f2 100644
--- a/azurelinuxagent/ga/update.py
+++ b/azurelinuxagent/ga/update.py
@@ -31,12 +31,10 @@ import uuid
 import zipfile
 from datetime import datetime, timedelta
 
-import azurelinuxagent.common.conf as conf
-import azurelinuxagent.common.logger as logger
+from azurelinuxagent.common import conf
+from azurelinuxagent.common import logger
 from azurelinuxagent.common.protocol.imds import get_imds_client
-import azurelinuxagent.common.utils.fileutil as fileutil
-import azurelinuxagent.common.utils.restutil as restutil
-import azurelinuxagent.common.utils.textutil as textutil
+from azurelinuxagent.common.utils import fileutil, restutil, textutil
 from azurelinuxagent.common.agent_supported_feature import get_supported_feature_by_name, SupportedFeatureNames
 from azurelinuxagent.common.cgroupconfigurator import CGroupConfigurator
 from azurelinuxagent.common.event import add_event, initialize_event_logger_vminfo_common_parameters, \
@@ -45,6 +43,7 @@ from azurelinuxagent.common.exception import ResourceGoneError, UpdateError, Exi
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.osutil import get_osutil, systemd
 from azurelinuxagent.common.persist_firewall_rules import PersistFirewallRulesHandler
+from azurelinuxagent.common.protocol.goal_state import GoalStateSource
 from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol, VmSettingsNotSupported
 from azurelinuxagent.common.protocol.restapi import VMAgentUpdateStatus, VMAgentUpdateStatuses, ExtHandlerPackageList, \
     VERSION_0
@@ -576,7 +575,7 @@ class UpdateHandler(object):
             # The call to get_latest_agent_greater_than_daemon() also finds all agents in directory and sets the self.agents property.
             # This state is used to find the GuestAgent object with the current version later if requested version is available in last GS.
             available_agent = self.get_latest_agent_greater_than_daemon()
-            requested_version, _ = self.__get_requested_version_and_manifest_from_last_gs(protocol)
+            requested_version, _ = self.__get_requested_version_and_agent_family_from_last_gs()
             if requested_version is not None:
                 # If requested version specified, upgrade/downgrade to the specified version instantly as this is
                 # driven by the goal state (as compared to the agent periodically checking for new upgrades every hour)
@@ -662,7 +661,7 @@ class UpdateHandler(object):
         except Exception as exception:
             logger.warn("Error removing legacy history files: {0}", ustr(exception))
 
-    def __get_vmagent_update_status(self, protocol, goal_state_changed):
+    def __get_vmagent_update_status(self, goal_state_changed):
         """
         This function gets the VMAgent update status as per the last GoalState.
         Returns: None if the last GS does not ask for requested version else VMAgentUpdateStatus
@@ -673,7 +672,7 @@ class UpdateHandler(object):
         update_status = None
 
         try:
-            requested_version, manifest = self.__get_requested_version_and_manifest_from_last_gs(protocol)
+            requested_version, manifest = self.__get_requested_version_and_agent_family_from_last_gs()
             if manifest is None and goal_state_changed:
                 logger.info("Unable to report update status as no matching manifest found for family: {0}".format(
                     conf.get_autoupdate_gafamily()))
@@ -699,7 +698,7 @@ class UpdateHandler(object):
         return update_status
 
     def _report_status(self, exthandlers_handler):
-        vm_agent_update_status = self.__get_vmagent_update_status(exthandlers_handler.protocol, self._processing_new_extensions_goal_state())
+        vm_agent_update_status = self.__get_vmagent_update_status(self._processing_new_extensions_goal_state())
         # report_ext_handlers_status does its own error handling and returns None if an error occurred
         vm_status = exthandlers_handler.report_ext_handlers_status(
             goal_state_changed=self._processing_new_extensions_goal_state(),
@@ -1068,22 +1067,21 @@ class UpdateHandler(object):
                 str(e))
         return
 
-    @staticmethod
-    def __get_requested_version_and_manifest_from_last_gs(protocol):
+    def __get_requested_version_and_agent_family_from_last_gs(self):
         """
         Get the requested version and corresponding manifests from last GS if supported
         Returns: (Requested Version, Manifest) if supported and available
                  (None, None) if no manifests found in the last GS
                  (None, manifest) if not supported or not specified in GS
         """
-        family = conf.get_autoupdate_gafamily()
-        manifest_list, _ = protocol.get_vmagent_manifests()
-        manifests = [m for m in manifest_list if m.family == family and len(m.uris) > 0]
-        if len(manifests) == 0:
+        family_name = conf.get_autoupdate_gafamily()
+        agent_families = self._goal_state.extensions_goal_state.agent_families
+        agent_families = [m for m in agent_families if m.name == family_name and len(m.uris) > 0]
+        if len(agent_families) == 0:
             return None, None
-        if conf.get_enable_ga_versioning() and manifests[0].is_requested_version_specified:
-            return manifests[0].requested_version, manifests[0]
-        return None, manifests[0]
+        if conf.get_enable_ga_versioning() and agent_families[0].is_requested_version_specified:
+            return agent_families[0].requested_version, agent_families[0]
+        return None, agent_families[0]
 
     def _download_agent_if_upgrade_available(self, protocol, base_version=CURRENT_VERSION):
         """
@@ -1130,18 +1128,18 @@ class UpdateHandler(object):
                 return False
             return True
 
-        family = conf.get_autoupdate_gafamily()
+        agent_family_name = conf.get_autoupdate_gafamily()
         gs_updated = False
         daemon_version = self.__get_daemon_version_for_update()
         try:
             # Fetch the agent manifests from the latest Goal State
             goal_state_id = self._goal_state.extensions_goal_state.id
             gs_updated = self._processing_new_extensions_goal_state()
-            requested_version, manifest = self.__get_requested_version_and_manifest_from_last_gs(protocol)
-            if manifest is None:
+            requested_version, agent_family = self.__get_requested_version_and_agent_family_from_last_gs()
+            if agent_family is None:
                 logger.verbose(
                     u"No manifest links found for agent family: {0} for goal state {1}, skipping update check".format(
-                        family, goal_state_id))
+                        agent_family_name, goal_state_id))
                 return False
         except Exception as err:
             # If there's some issues in fetching the agent manifests, report it only on goal state change
@@ -1166,7 +1164,7 @@ class UpdateHandler(object):
                 return False
 
             logger.info("No requested version specified, checking for all versions for agent update (family: {0})",
-                        family)
+                        agent_family_name)
             self.last_attempt_time = now
 
         try:
@@ -1183,7 +1181,8 @@ class UpdateHandler(object):
                 logger.info(msg)
                 add_event(AGENT_NAME, op=WALAEventOperation.AgentUpgrade, is_success=True, message=msg)
             else:
-                pkg_list = protocol.get_vmagent_pkgs(manifest)
+                agent_manifest = self._goal_state.fetch_agent_manifest(agent_family.name, agent_family.uris)
+                pkg_list = agent_manifest.pkg_list
                 packages_to_download = pkg_list.versions
 
             # Verify the requested version is in GA family manifest (if specified)
@@ -1202,7 +1201,7 @@ class UpdateHandler(object):
             # Set the agents to those available for download at least as current as the existing agent
             # or to the requested version (if specified)
             host = self._get_host_plugin(protocol=protocol)
-            agents_to_download = [GuestAgent(pkg=pkg, host=host) for pkg in packages_to_download]
+            agents_to_download = [GuestAgent(is_fast_track_goal_state=self._goal_state.extensions_goal_state.source == GoalStateSource.FastTrack, pkg=pkg, host=host) for pkg in packages_to_download]
 
             # Filter out the agents that were downloaded/extracted successfully. If the agent was not installed properly,
             # we delete the directory and the zip package from the filesystem
@@ -1468,7 +1467,16 @@ class UpdateHandler(object):
 
 
 class GuestAgent(object):
-    def __init__(self, path=None, pkg=None, host=None):
+    def __init__(self, path=None, pkg=None, is_fast_track_goal_state=False, host=None):
+        """
+        If 'path' is given, the object is initialized to the version installed under that path.
+
+        If 'pkg' is given, the version specified in the package information is downloaded and the object is
+        initialized to that version.
+
+        'is_fast_track_goal_state' and 'host' are using only when a package is downloaded.
+        """
+        self._is_fast_track_goal_state = is_fast_track_goal_state
         self.pkg = pkg
         self.host = host
         version = None
@@ -1614,7 +1622,7 @@ class GuestAgent(object):
                 else:
                     logger.verbose("Using host plugin as default channel")
 
-                uri, headers = self.host.get_artifact_request(uri, self.host.manifest_uri)
+                uri, headers = self.host.get_artifact_request(uri, use_verify_header=self._is_fast_track_goal_state, artifact_manifest_url=self.host.manifest_uri)
                 try:
                     if self._fetch(uri, headers=headers, use_proxy=False, retry_codes=restutil.HGAP_GET_EXTENSION_ARTIFACT_RETRY_CODES):
                         if not HostPluginProtocol.is_default_channel:
@@ -1692,7 +1700,7 @@ class GuestAgent(object):
             try:
                 manifests = json.load(manifest_file)
             except Exception as e:
-                msg = u"Agent {0} has a malformed {1}".format(self.name, AGENT_MANIFEST_FILE)
+                msg = u"Agent {0} has a malformed {1} ({2})".format(self.name, AGENT_MANIFEST_FILE, ustr(e))
                 raise UpdateError(msg)
             if type(manifests) is list:
                 if len(manifests) <= 0:
diff --git a/ci/3.6.pylintrc b/ci/3.6.pylintrc
index 7b0712d9..fcbae938 100644
--- a/ci/3.6.pylintrc
+++ b/ci/3.6.pylintrc
@@ -3,18 +3,22 @@
 [MESSAGES CONTROL]
 
 disable=C, # (C) convention, for programming standard violation
+    broad-except, # (W0703): *Catching too general exception %s*
     consider-using-dict-comprehension, # (R1717): *Consider using a dictionary comprehension*
     consider-using-in, # (R1714): *Consider merging these comparisons with "in" to %r*
     consider-using-set-comprehension, # (R1718): *Consider using a set comprehension*
     consider-using-with, # (R1732): *Emitted if a resource-allocating assignment or call may be replaced by a 'with' block*
     duplicate-code, # (R0801): *Similar lines in %s files*
-    no-init, # (W0232): Class has no __init__ method
+    fixme, # Used when a warning note as FIXME or TODO is detected
     no-else-break, # (R1723): *Unnecessary "%s" after "break"*
     no-else-continue, # (R1724): *Unnecessary "%s" after "continue"*
     no-else-raise, # (R1720): *Unnecessary "%s" after "raise"*
     no-else-return, # (R1705): *Unnecessary "%s" after "return"*
+    no-init, # (W0232): Class has no __init__ method
     no-self-use, # (R0201): *Method could be a function*
     protected-access, # (W0212): Access to a protected member of a client class
+    raise-missing-from, # (W0707): *Consider explicitly re-raising using the 'from' keyword*
+    redundant-u-string-prefix, # The u prefix for strings is no longer necessary in Python >=3.0
     simplifiable-if-expression, # (R1719): *The if expression can be replaced with %s*
     simplifiable-if-statement, # (R1703): *The if statement can be replaced with %s*
     super-with-arguments, # (R1725): *Consider using Python 3 style super() without arguments*
@@ -29,9 +33,8 @@ disable=C, # (C) convention, for programming standard violation
     too-many-public-methods, # (R0904): *Too many public methods (%s/%s)*
     too-many-return-statements, # (R0911): *Too many return statements (%s/%s)*
     too-many-statements, # (R0915): *Too many statements (%s/%s)*
+    unspecified-encoding, # (W1514): Using open without explicitly specifying an encoding
+    use-a-generator, # (R1729): *Use a generator instead '%s(%s)'*
     useless-object-inheritance, # (R0205): *Class %r inherits from object, can be safely removed from bases in python3*
     useless-return, # (R1711): *Useless return at end of function or method*
-    use-a-generator, # (R1729): *Use a generator instead '%s(%s)'*
-    broad-except, # (W0703): *Catching too general exception %s*
-    raise-missing-from, # (W0707): *Consider explicitly re-raising using the 'from' keyword*
-    fixme, # Used when a warning note as FIXME or TODO is detected
+


Test patch:
diff --git a/tests/ga/test_extension.py b/tests/ga/test_extension.py
index ca502f3c..1f261ee2 100644
--- a/tests/ga/test_extension.py
+++ b/tests/ga/test_extension.py
@@ -1361,7 +1361,7 @@ class TestExtension_Deprecated(TestExtensionBase):
     def test_ext_handler_download_failure_permanent_ProtocolError(self, mock_add_event, mock_error_state, *args):
         test_data = mockwiredata.WireProtocolData(mockwiredata.DATA_FILE)
         exthandlers_handler, protocol = self._create_mock(test_data, *args)  # pylint: disable=no-value-for-parameter
-        protocol.get_ext_handler_pkgs = Mock(side_effect=ProtocolError)
+        protocol.get_goal_state().fetch_extension_manifest = Mock(side_effect=ProtocolError)
 
         mock_error_state.return_value = True
 
@@ -2496,7 +2496,7 @@ class TestExtensionSequencing(AgentTestCase):
         """
         Creates extensions with the given dependencyLevel
         """
-        handler_map = dict()
+        handler_map = {}
         all_handlers = []
         for handler_name, level in dependency_levels:
             if handler_map.get(handler_name) is None:
@@ -3308,7 +3308,7 @@ class TestAdditionalLocationsExtensions(AgentTestCase):
             wire._DOWNLOAD_TIMEOUT = datetime.timedelta(minutes=0)
             try:
                 with self.assertRaises(ExtensionDownloadError):
-                    protocol.client.fetch_manifest(ext_handlers[0].manifest_uris)
+                    protocol.client.fetch_manifest(ext_handlers[0].manifest_uris, use_verify_header=False)
             finally:
                 wire._DOWNLOAD_TIMEOUT = download_timeout
 
diff --git a/tests/ga/test_exthandlers_download_extension.py b/tests/ga/test_exthandlers_download_extension.py
index e3212ae1..3a968388 100644
--- a/tests/ga/test_exthandlers_download_extension.py
+++ b/tests/ga/test_exthandlers_download_extension.py
@@ -10,6 +10,8 @@ from azurelinuxagent.common.exception import ExtensionDownloadError, ExtensionEr
 from azurelinuxagent.common.protocol.restapi import Extension, ExtHandlerPackage
 from azurelinuxagent.common.protocol.wire import WireProtocol
 from azurelinuxagent.ga.exthandlers import ExtHandlerInstance, ExtHandlerState
+from tests.protocol import mockwiredata
+from tests.protocol.mocks import mock_wire_protocol
 from tests.tools import AgentTestCase, patch, Mock
 
 
@@ -39,6 +41,12 @@ class DownloadExtensionTestCase(AgentTestCase):
         protocol.client.get_host_plugin = Mock()
         protocol.client.get_artifact_request = Mock(return_value=(None, None))
 
+        # create a dummy goal state, since downloads are done via the GoalState class
+        with mock_wire_protocol(mockwiredata.DATA_FILE) as p:
+            goal_state = p.get_goal_state()
+            goal_state._wire_client = protocol.client
+            protocol.client._goal_state = goal_state
+
         self.pkg = ExtHandlerPackage()
         self.pkg.uris = [
             'https://zrdfepirv2cy4prdstr00a.blob.core.windows.net/f72653efd9e349ed9842c8b99e4c1712-foobar/Microsoft.CPlat.Core__RunCommandLinux__1.0.0',
diff --git a/tests/ga/test_update.py b/tests/ga/test_update.py
index edcf669d..695b6d57 100644
--- a/tests/ga/test_update.py
+++ b/tests/ga/test_update.py
@@ -33,7 +33,7 @@ from azurelinuxagent.common.future import ustr, httpclient
 from azurelinuxagent.common.persist_firewall_rules import PersistFirewallRulesHandler
 from azurelinuxagent.common.protocol.hostplugin import URI_FORMAT_GET_API_VERSIONS, HOST_PLUGIN_PORT, \
     URI_FORMAT_GET_EXTENSION_ARTIFACT, HostPluginProtocol
-from azurelinuxagent.common.protocol.restapi import VMAgentManifest, \
+from azurelinuxagent.common.protocol.restapi import VMAgentFamily, \
     ExtHandlerPackage, ExtHandlerPackageList, Extension, VMStatus, ExtHandlerStatus, ExtensionStatus, \
     VMAgentUpdateStatuses
 from azurelinuxagent.common.protocol.util import ProtocolUtil
@@ -194,7 +194,7 @@ class UpdateTestCase(AgentTestCaseWithGetVmSizeMock):
         shutil.move(src_bin, dst_bin)
 
     def agents(self):
-        return [GuestAgent(path=path) for path in self.agent_dirs()]
+        return [GuestAgent(is_fast_track_goal_state=False, path=path) for path in self.agent_dirs()]
 
     def agent_count(self):
         return len(self.agent_dirs())
@@ -313,7 +313,7 @@ class UpdateTestCase(AgentTestCaseWithGetVmSizeMock):
             shutil.copytree(from_path, to_path)
             self.rename_agent_bin(to_path, dst_v)
             if not is_available:
-                GuestAgent(to_path).mark_failure(is_fatal=True)
+                GuestAgent(is_fast_track_goal_state=False, path=to_path).mark_failure(is_fatal=True)
         return dst_v
 
 
@@ -411,7 +411,7 @@ class TestGuestAgent(UpdateTestCase):
 
         self.expand_agents()
 
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertNotEqual(None, agent)
         self.assertEqual(self._get_agent_name(), agent.name)
         self.assertEqual(self._get_agent_version(), agent.version)
@@ -436,7 +436,7 @@ class TestGuestAgent(UpdateTestCase):
     def test_clear_error(self, mock_downloaded):  # pylint: disable=unused-argument
         self.expand_agents()
 
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         agent.mark_failure(is_fatal=True)
 
         self.assertTrue(agent.error.last_failure > 0.0)
@@ -453,7 +453,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_is_available(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
 
         self.assertFalse(agent.is_available)
         agent._unpack()
@@ -465,7 +465,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_is_blacklisted(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(agent.is_blacklisted)
 
         agent._unpack()
@@ -481,7 +481,7 @@ class TestGuestAgent(UpdateTestCase):
     def test_resource_gone_error_not_blacklisted(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
         try:
             mock_downloaded.side_effect = ResourceGoneError()
-            agent = GuestAgent(path=self.agent_path)
+            agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
             self.assertFalse(agent.is_blacklisted)
         except ResourceGoneError:
             pass
@@ -493,7 +493,7 @@ class TestGuestAgent(UpdateTestCase):
     def test_ioerror_not_blacklisted(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
         try:
             mock_downloaded.side_effect = IOError()
-            agent = GuestAgent(path=self.agent_path)
+            agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
             self.assertFalse(agent.is_blacklisted)
         except IOError:
             pass
@@ -503,7 +503,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_is_downloaded(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(agent.is_downloaded)
         agent._unpack()
         self.assertTrue(agent.is_downloaded)
@@ -511,7 +511,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_mark_failure(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
 
         agent.mark_failure()
         self.assertEqual(1, agent.error.failure_count)
@@ -523,7 +523,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_unpack(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(os.path.isdir(agent.get_agent_dir()))
         agent._unpack()
         self.assertTrue(os.path.isdir(agent.get_agent_dir()))
@@ -532,7 +532,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_unpack_fail(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(os.path.isdir(agent.get_agent_dir()))
         os.remove(agent.get_agent_pkg_path())
         self.assertRaises(UpdateError, agent._unpack)
@@ -540,7 +540,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_load_manifest(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         agent._unpack()
         agent._load_manifest()
         self.assertEqual(agent.manifest.get_enable_command(),
@@ -549,7 +549,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_load_manifest_missing(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(os.path.isdir(agent.get_agent_dir()))
         agent._unpack()
         os.remove(agent.get_agent_manifest_path())
@@ -558,7 +558,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_load_manifest_is_empty(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(os.path.isdir(agent.get_agent_dir()))
         agent._unpack()
         self.assertTrue(os.path.isfile(agent.get_agent_manifest_path()))
@@ -570,7 +570,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_loaded")
     def test_load_manifest_is_malformed(self, mock_loaded, mock_downloaded):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertFalse(os.path.isdir(agent.get_agent_dir()))
         agent._unpack()
         self.assertTrue(os.path.isfile(agent.get_agent_manifest_path()))
@@ -580,7 +580,7 @@ class TestGuestAgent(UpdateTestCase):
         self.assertRaises(UpdateError, agent._load_manifest)
 
     def test_load_error(self):
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         agent.error = None
 
         agent._load_error()
@@ -598,7 +598,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
         agent._download()
 
         self.assertTrue(os.path.isfile(agent.get_agent_pkg_path()))
@@ -614,7 +614,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertRaises(UpdateError, agent._download)
         self.assertFalse(os.path.isfile(agent.get_agent_pkg_path()))
@@ -640,7 +640,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(ext_uri)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
         agent.host = mock_host
 
         # ensure fallback fails gracefully, no http
@@ -689,7 +689,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertTrue(os.path.isfile(agent.get_agent_manifest_path()))
         self.assertTrue(agent.is_downloaded)
@@ -701,7 +701,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertFalse(agent.is_blacklisted, "The agent should not be blacklisted if unable to unpack/download")
         self.assertFalse(os.path.exists(agent.get_agent_dir()), "Agent directory should be cleaned up")
@@ -714,7 +714,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertFalse(agent.is_blacklisted, "The agent should not be blacklisted if unable to unpack/download")
         self.assertFalse(os.path.exists(agent.get_agent_dir()), "Agent directory should be cleaned up")
@@ -728,7 +728,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertFalse(agent.is_blacklisted, "The agent should not be blacklisted if unable to unpack/download")
         self.assertFalse(os.path.exists(agent.get_agent_dir()), "Agent directory should be cleaned up")
@@ -738,7 +738,7 @@ class TestGuestAgent(UpdateTestCase):
     @patch("azurelinuxagent.ga.update.GuestAgent._unpack")
     @patch("azurelinuxagent.ga.update.GuestAgent._load_manifest")
     def test_ensure_download_skips_blacklisted(self, mock_manifest, mock_unpack, mock_download):  # pylint: disable=unused-argument
-        agent = GuestAgent(path=self.agent_path)
+        agent = GuestAgent(is_fast_track_goal_state=False, path=self.agent_path)
         self.assertEqual(0, mock_download.call_count)
 
         agent.clear_error()
@@ -747,7 +747,7 @@ class TestGuestAgent(UpdateTestCase):
 
         pkg = ExtHandlerPackage(version=str(self._get_agent_version()))
         pkg.uris.append(None)
-        agent = GuestAgent(pkg=pkg)
+        agent = GuestAgent(is_fast_track_goal_state=False, pkg=pkg)
 
         self.assertEqual(1, agent.error.failure_count)
         self.assertTrue(agent.error.was_fatal)
@@ -764,6 +764,9 @@ class TestUpdate(UpdateTestCase):
         protocol = Mock()
         self.update_handler.protocol_util = Mock()
         self.update_handler.protocol_util.get_protocol = Mock(return_value=protocol)
+        self.update_handler._goal_state = Mock()
+        self.update_handler._goal_state.extensions_goal_state = Mock()
+        self.update_handler._goal_state.extensions_goal_state.source = "Fabric"
 
         # Since ProtocolUtil is a singleton per thread, we need to clear it to ensure that the test cases do not reuse
         # a previous state
@@ -951,7 +954,7 @@ class TestUpdate(UpdateTestCase):
     def test_filter_blacklisted_agents(self):
         self.prepare_agents()
 
-        self.update_handler._set_and_sort_agents([GuestAgent(path=path) for path in self.agent_dirs()])
+        self.update_handler._set_and_sort_agents([GuestAgent(is_fast_track_goal_state=False, path=path) for path in self.agent_dirs()])
         self.assertEqual(len(self.agent_dirs()), len(self.update_handler.agents))
 
         kept_agents = self.update_handler.agents[::2]
@@ -1022,7 +1025,7 @@ class TestUpdate(UpdateTestCase):
 
         latest_version = self.prepare_agents(count=self.agent_count() + 1, is_available=False)
         latest_path = os.path.join(self.tmp_dir, "{0}-{1}".format(AGENT_NAME, latest_version))
-        self.assertFalse(GuestAgent(latest_path).is_available)
+        self.assertFalse(GuestAgent(is_fast_track_goal_state=False, path=latest_path).is_available)
 
         latest_agent = self.update_handler.get_latest_agent_greater_than_daemon()
         self.assertTrue(latest_agent.version < latest_version)
@@ -1297,14 +1300,14 @@ class TestUpdate(UpdateTestCase):
     def test_set_agents_sets_agents(self):
         self.prepare_agents()
 
-        self.update_handler._set_and_sort_agents([GuestAgent(path=path) for path in self.agent_dirs()])
+        self.update_handler._set_and_sort_agents([GuestAgent(is_fast_track_goal_state=False, path=path) for path in self.agent_dirs()])
         self.assertTrue(len(self.update_handler.agents) > 0)
         self.assertEqual(len(self.agent_dirs()), len(self.update_handler.agents))
 
     def test_set_agents_sorts_agents(self):
         self.prepare_agents()
 
-        self.update_handler._set_and_sort_agents([GuestAgent(path=path) for path in self.agent_dirs()])
+        self.update_handler._set_and_sort_agents([GuestAgent(is_fast_track_goal_state=False, path=path) for path in self.agent_dirs()])
 
         v = FlexibleVersion("100000")
         for a in self.update_handler.agents:
@@ -1370,7 +1373,7 @@ class TestUpdate(UpdateTestCase):
         with mock_wire_protocol(data_file) as protocol:
             self.update_handler.protocol_util = protocol
             with patch('azurelinuxagent.common.logger.warn') as mock_logger:
-                with patch('tests.ga.test_update.ProtocolMock.get_vmagent_pkgs', side_effect=ProtocolError):
+                with patch('azurelinuxagent.common.protocol.goal_state.GoalState.fetch_agent_manifest', side_effect=ProtocolError):
                     self.assertFalse(self.update_handler._download_agent_if_upgrade_available(protocol, base_version=CURRENT_VERSION))
                     self.assertEqual(0, mock_logger.call_count)
 
@@ -2522,15 +2525,42 @@ class ChildMock(Mock):
         self.wait = Mock(return_value=return_value, side_effect=side_effect)
 
 
-class ExtensionsGoalStateMock(object):
-    def __init__(self, identifier):
-        self.id = identifier
-
-
 class GoalStateMock(object):
-    def __init__(self, incarnation):
+    def __init__(self, incarnation, family, versions):
+        if versions is None:
+            versions = []
+
         self.incarnation = incarnation
-        self.extensions_goal_state = ExtensionsGoalStateMock(incarnation)
+        self.extensions_goal_state = Mock()
+        self.extensions_goal_state.id = incarnation
+        self.extensions_goal_state.agent_families = GoalStateMock._create_agent_families(family, versions)
+
+        agent_manifest = Mock()
+        agent_manifest.pkg_list = GoalStateMock._create_packages(versions)
+        self.fetch_agent_manifest = Mock(return_value=agent_manifest)
+
+    @staticmethod
+    def _create_agent_families(family, versions):
+        families = []
+
+        if len(versions) > 0 and family is not None:
+            manifest = VMAgentFamily(name=family)
+            for i in range(0, 10):
+                manifest.uris.append("https://nowhere.msft/agent/{0}".format(i))
+            families.append(manifest)
+
+        return families
+
+    @staticmethod
+    def _create_packages(versions):
+        packages = ExtHandlerPackageList()
+        for version in versions:
+            package = ExtHandlerPackage(str(version))
+            for i in range(0, 5):
+                package_uri = "https://nowhere.msft/agent_pkg/{0}".format(i)
+                package.uris.append(package_uri)
+            packages.versions.append(package)
+        return packages
 
 
 class ProtocolMock(object):
@@ -2538,63 +2568,22 @@ class ProtocolMock(object):
         self.family = family
         self.client = client
         self.call_counts = {
-            "get_vmagent_manifests": 0,
-            "get_vmagent_pkgs": 0,
             "update_goal_state": 0
         }
-        self._goal_state = GoalStateMock(etag)
+        self._goal_state = GoalStateMock(etag, family, versions)
         self.goal_state_is_stale = False
         self.etag = etag
         self.versions = versions if versions is not None else []
-        self.create_manifests()
-        self.create_packages()
 
     def emulate_stale_goal_state(self):
         self.goal_state_is_stale = True
 
-    def create_manifests(self):
-        self.agent_manifests = []
-        if len(self.versions) <= 0:
-            return
-
-        if self.family is not None:
-            manifest = VMAgentManifest(family=self.family)
-            for i in range(0, 10):
-                manifest.uris.append("https://nowhere.msft/agent/{0}".format(i))
-            self.agent_manifests.append(manifest)
-
-    def create_packages(self):
-        self.agent_packages = ExtHandlerPackageList()
-        if len(self.versions) <= 0:
-            return
-
-        for version in self.versions:
-            package = ExtHandlerPackage(str(version))
-            for i in range(0, 5):
-                package_uri = "https://nowhere.msft/agent_pkg/{0}".format(i)
-                package.uris.append(package_uri)
-            self.agent_packages.versions.append(package)
-
     def get_protocol(self):
         return self
 
     def get_goal_state(self):
         return self._goal_state
 
-    def get_vmagent_manifests(self):
-        self.call_counts["get_vmagent_manifests"] += 1
-        if self.goal_state_is_stale:
-            self.goal_state_is_stale = False
-            raise ResourceGoneError()
-        return self.agent_manifests, self.etag
-
-    def get_vmagent_pkgs(self, manifest):  # pylint: disable=unused-argument
-        self.call_counts["get_vmagent_pkgs"] += 1
-        if self.goal_state_is_stale:
-            self.goal_state_is_stale = False
-            raise ResourceGoneError()
-        return self.agent_packages
-
     def update_goal_state(self):
         self.call_counts["update_goal_state"] += 1
 
diff --git a/tests/protocol/test_extensions_goal_state_from_vm_settings.py b/tests/protocol/test_extensions_goal_state_from_vm_settings.py
index 8cdfa81b..fb97a075 100644
--- a/tests/protocol/test_extensions_goal_state_from_vm_settings.py
+++ b/tests/protocol/test_extensions_goal_state_from_vm_settings.py
@@ -2,6 +2,7 @@
 # Licensed under the Apache License.
 import json
 
+from azurelinuxagent.common.protocol.goal_state import GoalState
 from azurelinuxagent.common.protocol.extensions_goal_state import GoalStateChannel
 from azurelinuxagent.common.protocol.extensions_goal_state_from_vm_settings import _CaseFoldedDict
 from tests.protocol.mocks import mockwiredata, mock_wire_protocol
@@ -28,12 +29,12 @@ class ExtensionsGoalStateFromVmSettingsTestCase(AgentTestCase):
             # for the rest of the attributes, we check only 1 item in each container (but check the length of the container)
             #
 
-            # agent manifests
-            self.assertEqual(2, len(extensions_goal_state.agent_manifests), "Incorrect number of agent manifests. Got: {0}".format(extensions_goal_state.agent_manifests))
-            self.assertEqual("Prod", extensions_goal_state.agent_manifests[0].family, "Incorrect agent family.")
-            self.assertEqual(2, len(extensions_goal_state.agent_manifests[0].uris), "Incorrect number of uris. Got: {0}".format(extensions_goal_state.agent_manifests[0].uris))
+            # agent families
+            self.assertEqual(2, len(extensions_goal_state.agent_families), "Incorrect number of agent families. Got: {0}".format(extensions_goal_state.agent_families))
+            self.assertEqual("Prod", extensions_goal_state.agent_families[0].name, "Incorrect agent family.")
+            self.assertEqual(2, len(extensions_goal_state.agent_families[0].uris), "Incorrect number of uris. Got: {0}".format(extensions_goal_state.agent_families[0].uris))
             expected = "https://zrdfepirv2cdm03prdstr01a.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent_Prod_uscentraleuap_manifest.xml"
-            self.assertEqual(expected, extensions_goal_state.agent_manifests[0].uris[0], "Unexpected URI for the agent manifest.")
+            self.assertEqual(expected, extensions_goal_state.agent_families[0].uris[0], "Unexpected URI for the agent manifest.")
 
             # extensions
             self.assertEqual(5, len(extensions_goal_state.extensions), "Incorrect number of extensions. Got: {0}".format(extensions_goal_state.extensions))
@@ -49,16 +50,18 @@ class ExtensionsGoalStateFromVmSettingsTestCase(AgentTestCase):
 
     def test_it_should_parse_requested_version_properly(self):
         with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
-            manifests, _ = protocol.get_vmagent_manifests()
-            for manifest in manifests:
-                self.assertEqual(manifest.requested_version_string, "0.0.0.0", "Version should be None")
+            goal_state = GoalState(protocol.client)
+            families = goal_state.extensions_goal_state.agent_families
+            for family in families:
+                self.assertEqual(family.requested_version_string, "0.0.0.0", "Version should be None")
 
         data_file = mockwiredata.DATA_FILE_VM_SETTINGS.copy()
         data_file["vm_settings"] = "hostgaplugin/vm_settings-requested_version.json"
         with mock_wire_protocol(data_file) as protocol:
-            manifests, _ = protocol.get_vmagent_manifests()
-            for manifest in manifests:
-                self.assertEqual(manifest.requested_version_string, "9.9.9.9", "Version should be 9.9.9.9")
+            goal_state = GoalState(protocol.client)
+            families = goal_state.extensions_goal_state.agent_families
+            for family in families:
+                self.assertEqual(family.requested_version_string, "9.9.9.9", "Version should be 9.9.9.9")
 
     def test_it_should_parse_missing_status_upload_blob_as_none(self):
         data_file = mockwiredata.DATA_FILE_VM_SETTINGS.copy()
@@ -74,8 +77,8 @@ class ExtensionsGoalStateFromVmSettingsTestCase(AgentTestCase):
         data_file["vm_settings"] = "hostgaplugin/vm_settings-no_manifests.json"
         with mock_wire_protocol(data_file) as protocol:
             extensions_goal_state = protocol.get_goal_state().extensions_goal_state
-            self.assertEqual(1, len(extensions_goal_state.agent_manifests), "Expected exactly one agent manifest. Got: {0}".format(extensions_goal_state.agent_manifests))
-            self.assertListEqual([], extensions_goal_state.agent_manifests[0].uris, "Expected an empty list of agent manifests")
+            self.assertEqual(1, len(extensions_goal_state.agent_families), "Expected exactly one agent manifest. Got: {0}".format(extensions_goal_state.agent_families))
+            self.assertListEqual([], extensions_goal_state.agent_families[0].uris, "Expected an empty list of agent manifests")
 
     def test_it_should_parse_missing_extension_manifests_as_empty(self):
         data_file = mockwiredata.DATA_FILE_VM_SETTINGS.copy()
diff --git a/tests/protocol/test_hostplugin.py b/tests/protocol/test_hostplugin.py
index 9f96f7d5..42d8579d 100644
--- a/tests/protocol/test_hostplugin.py
+++ b/tests/protocol/test_hostplugin.py
@@ -23,9 +23,7 @@ import os.path
 import sys
 import unittest
 
-import azurelinuxagent.common.protocol.hostplugin as hostplugin
-import azurelinuxagent.common.protocol.restapi as restapi
-import azurelinuxagent.common.protocol.wire as wire
+from azurelinuxagent.common.protocol import hostplugin, restapi, wire
 from azurelinuxagent.common import conf
 from azurelinuxagent.common.errorstate import ErrorState
 from azurelinuxagent.common.exception import HttpError, ResourceGoneError, ProtocolError
@@ -609,7 +607,7 @@ class TestHostPlugin(HttpRequestPredicates, AgentTestCase):
             self.assertTrue(host_client.health_service is not None)
 
             with patch.object(wire.HostPluginProtocol, "get_api_versions", return_value=api_versions) as patch_get:  # pylint: disable=unused-variable
-                actual_url, actual_headers = host_client.get_artifact_request(sas_url)
+                actual_url, actual_headers = host_client.get_artifact_request(sas_url, use_verify_header=False)
                 self.assertTrue(host_client.is_initialized)
                 self.assertFalse(host_client.api_versions is None)
                 self.assertEqual(expected_url, actual_url)
diff --git a/tests/protocol/test_wire.py b/tests/protocol/test_wire.py
index e96cf598..0cfaa95f 100644
--- a/tests/protocol/test_wire.py
+++ b/tests/protocol/test_wire.py
@@ -91,7 +91,7 @@ class TestWireProtocol(AgentTestCase, HttpRequestPredicates):
             protocol.get_certs()
             ext_handlers = protocol.get_goal_state().extensions_goal_state.extensions
             for ext_handler in ext_handlers:
-                protocol.get_ext_handler_pkgs(ext_handler)
+                protocol.get_goal_state().fetch_extension_manifest(ext_handler.name, ext_handler.manifest_uris)
 
             crt1 = os.path.join(self.tmp_dir,
                                 '38B85D88F03D1A8E1C671EB169274C09BC4D4703.crt')
@@ -471,9 +471,9 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
             ext_handlers_names = [ext_handler.name for ext_handler in extensions_goal_state.extensions]
             self.assertEqual(0, len(extensions_goal_state.extensions),
                              "Unexpected number of extension handlers in the extension config: [{0}]".format(ext_handlers_names))
-            vmagent_manifests = [manifest.family for manifest in extensions_goal_state.agent_manifests]
-            self.assertEqual(0, len(extensions_goal_state.agent_manifests),
-                             "Unexpected number of vmagent manifests in the extension config: [{0}]".format(vmagent_manifests))
+            vmagent_families = [manifest.name for manifest in extensions_goal_state.agent_families]
+            self.assertEqual(0, len(extensions_goal_state.agent_families),
+                             "Unexpected number of vmagent manifests in the extension config: [{0}]".format(vmagent_families))
             self.assertFalse(extensions_goal_state.on_hold,
                               "Extensions On Hold is expected to be False")
 
@@ -486,9 +486,9 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
             ext_handlers_names = [ext_handler.name for ext_handler in extensions_goal_state.extensions]
             self.assertEqual(1, len(extensions_goal_state.extensions),
                              "Unexpected number of extension handlers in the extension config: [{0}]".format(ext_handlers_names))
-            vmagent_manifests = [manifest.family for manifest in extensions_goal_state.agent_manifests]
-            self.assertEqual(2, len(extensions_goal_state.agent_manifests),
-                             "Unexpected number of vmagent manifests in the extension config: [{0}]".format(vmagent_manifests))
+            vmagent_families = [manifest.name for manifest in extensions_goal_state.agent_families]
+            self.assertEqual(2, len(extensions_goal_state.agent_families),
+                             "Unexpected number of vmagent manifests in the extension config: [{0}]".format(vmagent_families))
             self.assertEqual("https://test.blob.core.windows.net/vhds/test-cs12.test-cs12.test-cs12.status?sr=b&sp=rw"
                              "&se=9999-01-01&sk=key1&sv=2014-02-14&sig=hfRh7gzUE7sUtYwke78IOlZOrTRCYvkec4hGZ9zZzXo",
                              extensions_goal_state.status_upload_blob, "Unexpected value for status upload blob URI")
@@ -511,7 +511,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
         with mock_wire_protocol(mockwiredata.DATA_FILE, http_get_handler=http_get_handler) as protocol:
             HostPluginProtocol.is_default_channel = False
 
-            protocol.client.download_extension([extension_url], target_file)
+            protocol.client.download_extension([extension_url], target_file, use_verify_header=False)
 
             urls = protocol.get_tracked_urls()
             self.assertEqual(len(urls), 1, "Unexpected number of HTTP requests: [{0}]".format(urls))
@@ -533,7 +533,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
         with mock_wire_protocol(mockwiredata.DATA_FILE, http_get_handler=http_get_handler) as protocol:
             HostPluginProtocol.is_default_channel = False
 
-            protocol.client.download_extension([extension_url], target_file)
+            protocol.client.download_extension([extension_url], target_file, use_verify_header=False)
 
             urls = protocol.get_tracked_urls()
             self.assertEqual(len(urls), 2, "Unexpected number of HTTP requests: [{0}]".format(urls))
@@ -569,7 +569,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
 
                 protocol.set_http_handlers(http_get_handler=http_get_handler)
 
-                protocol.client.download_extension([extension_url], target_file)
+                protocol.client.download_extension([extension_url], target_file, use_verify_header=False)
 
                 urls = protocol.get_tracked_urls()
                 self.assertEqual(len(urls), 4, "Unexpected number of HTTP requests: [{0}]".format(urls))
@@ -602,7 +602,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
             protocol.set_http_handlers(http_get_handler=http_get_handler)
 
             with self.assertRaises(ExtensionDownloadError):
-                protocol.client.download_extension([extension_url], target_file)
+                protocol.client.download_extension([extension_url], target_file, use_verify_header=False)
 
             urls = protocol.get_tracked_urls()
             self.assertEqual(len(urls), 2, "Unexpected number of HTTP requests: [{0}]".format(urls))
@@ -625,7 +625,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
         with mock_wire_protocol(mockwiredata.DATA_FILE, http_get_handler=http_get_handler) as protocol:
             HostPluginProtocol.is_default_channel = False
 
-            manifest = protocol.client.fetch_manifest([manifest_url])
+            manifest = protocol.client.fetch_manifest([manifest_url], use_verify_header=False)
 
             urls = protocol.get_tracked_urls()
             self.assertEqual(manifest, manifest_xml, 'The expected manifest was not downloaded')
@@ -648,7 +648,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
             HostPluginProtocol.is_default_channel = False
 
             try:
-                manifest = protocol.client.fetch_manifest([manifest_url])
+                manifest = protocol.client.fetch_manifest([manifest_url], use_verify_header=False)
 
                 urls = protocol.get_tracked_urls()
                 self.assertEqual(manifest, manifest_xml, 'The expected manifest was not downloaded')
@@ -685,7 +685,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
                 protocol.client.get_host_plugin()
 
                 protocol.set_http_handlers(http_get_handler=http_get_handler)
-                manifest = protocol.client.fetch_manifest([manifest_url])
+                manifest = protocol.client.fetch_manifest([manifest_url], use_verify_header=False)
 
                 urls = protocol.get_tracked_urls()
                 self.assertEqual(manifest, manifest_xml)
@@ -719,7 +719,7 @@ class TestWireClient(HttpRequestPredicates, AgentTestCase):
             protocol.set_http_handlers(http_get_handler=http_get_handler)
 
             with self.assertRaises(ExtensionDownloadError):
-                protocol.client.fetch_manifest([manifest_url])
+                protocol.client.fetch_manifest([manifest_url], use_verify_header=False)
 
             urls = protocol.get_tracked_urls()
             self.assertEqual(len(urls), 4, "Unexpected number of HTTP requests: [{0}]".format(urls))