Repository: Azure/WALinuxAgent
Pull request #2542 (2022-04-05 15:09:49)

Skip outdated Fabric goal states when HGAP does not support vmSettings
If the agent is using FastTrack and the VM is migrated from a node where the HGAP supports vmSettings to a node where it doesn't we need to ensure that the agent does not execute an outdated Fabric goal state.

The HostPluginProtocol now keeps track of the timestamp of the most recent FastTrack goal state. If the fetch_vm_settings() method detects that support for vmSettings is no longer available, it raises  VmSettingsSupportStopped.

GoalState.update() handles that exception by restoring the current Fabric goal state. If the timestamp of this goal state is older than the most recent FastTrack goal state, it marks the GS as "outdated".

The UpdateHandler will not execute extensions if the GS is outdated.

The timestamp for the FastTrack goal state is persisted to /var/lib/waagent/fast_track.json. This is needed to handle agent restarts. The file is deleted once the agent reports status (at that point the agent will report that it does not support FastTrack and CRP will switch to Fabric.

The PR includes cleanup of related code. I'll add comments pointing out this cleanup.

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
> Merging [#2542](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (9b46f3a) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/f91bf787344dc06e653d4861c03741dbf968b7ee?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (f91bf78) will **increase** coverage by `0.04%`.
> The diff coverage is `89.51%`.

```diff
@@             Coverage Diff             @@
##           develop    #2542      +/-   ##
===========================================
+ Coverage    72.00%   72.04%   +0.04%     
===========================================
  Files          102      102              
  Lines        15229    15298      +69     
  Branches      2416     2422       +6     
===========================================
+ Hits         10965    11022      +57     
- Misses        3779     3787       +8     
- Partials       485      489       +4     
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/protocol/hostplugin.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9ob3N0cGx1Z2luLnB5) | `87.60% <86.51%> (-1.26%)` | :arrow_down: |
| [azurelinuxagent/common/protocol/goal\_state.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9nb2FsX3N0YXRlLnB5) | `95.25% <95.45%> (-0.05%)` | :arrow_down: |
| [...inuxagent/common/protocol/extensions\_goal\_state.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGUucHk=) | `74.46% <100.00%> (+1.53%)` | :arrow_up: |
| [...ol/extensions\_goal\_state\_from\_extensions\_config.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9leHRlbnNpb25zX2dvYWxfc3RhdGVfZnJvbV9leHRlbnNpb25zX2NvbmZpZy5weQ==) | `91.19% <100.00%> (+0.03%)` | :arrow_up: |
| [azurelinuxagent/ga/update.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL3VwZGF0ZS5weQ==) | `88.40% <100.00%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Last update [f91bf78...9b46f3a](https://codecov.io/gh/Azure/WALinuxAgent/pull/2542?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).


Patch:
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state.py b/azurelinuxagent/common/protocol/extensions_goal_state.py
index 454a1380..74c761ca 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state.py
@@ -53,6 +53,9 @@ class ExtensionsGoalState(object):
 
     NOTE: This is an abstract class. The corresponding concrete classes can be instantiated using the ExtensionsGoalStateFactory.
     """
+    def __init__(self):
+        self._is_outdated = False
+
     @property
     def id(self):
         """
@@ -61,6 +64,18 @@ class ExtensionsGoalState(object):
         """
         raise NotImplementedError()
 
+    @property
+    def is_outdated(self):
+        """
+        A goal state can be outdated if, for example, the VM Agent is using Fast Track and support for it stops (e.g. the VM is migrated
+        to a node with an older version of the HostGAPlugin) and now the Agent is fetching goal states via the WireServer.
+        """
+        return self._is_outdated
+
+    @is_outdated.setter
+    def is_outdated(self, value):
+        self._is_outdated = value
+
     @property
     def svd_sequence_number(self):
         raise NotImplementedError()
diff --git a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
index 98ab84cc..c7e01dd2 100644
--- a/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
+++ b/azurelinuxagent/common/protocol/extensions_goal_state_from_extensions_config.py
@@ -33,6 +33,7 @@ class ExtensionsGoalStateFromExtensionsConfig(ExtensionsGoalState):
     def __init__(self, incarnation, xml_text, wire_client):
         super(ExtensionsGoalStateFromExtensionsConfig, self).__init__()
         self._id = "incarnation_{0}".format(incarnation)
+        self._is_outdated = False
         self._incarnation = incarnation
         self._text = xml_text
         self._status_upload_blob = None
diff --git a/azurelinuxagent/common/protocol/goal_state.py b/azurelinuxagent/common/protocol/goal_state.py
index 7c4c7c8c..6d362438 100644
--- a/azurelinuxagent/common/protocol/goal_state.py
+++ b/azurelinuxagent/common/protocol/goal_state.py
@@ -23,11 +23,12 @@ import azurelinuxagent.common.conf as conf
 import azurelinuxagent.common.logger as logger
 from azurelinuxagent.common.AgentGlobals import AgentGlobals
 from azurelinuxagent.common.datacontract import set_properties
+from azurelinuxagent.common.event import add_event, WALAEventOperation
 from azurelinuxagent.common.exception import ProtocolError, ResourceGoneError
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.extensions_goal_state_factory import ExtensionsGoalStateFactory
 from azurelinuxagent.common.protocol.extensions_goal_state import VmSettingsParseError, GoalStateSource
-from azurelinuxagent.common.protocol.hostplugin import VmSettingsNotSupported
+from azurelinuxagent.common.protocol.hostplugin import VmSettingsNotSupported, VmSettingsSupportStopped
 from azurelinuxagent.common.protocol.restapi import Cert, CertList, RemoteAccessUser, RemoteAccessUsersList
 from azurelinuxagent.common.utils import fileutil, timeutil
 from azurelinuxagent.common.utils.archive import GoalStateHistory
@@ -134,9 +135,15 @@ class GoalState(object):
         incarnation, xml_text, xml_doc = GoalState._fetch_goal_state(self._wire_client)
         goal_state_updated = incarnation != self._incarnation
         if goal_state_updated:
-            logger.info('Fetched new goal state from the WireServer [incarnation {0}]', incarnation)
+            logger.info('Fetched a new incarnation for the WireServer goal state [incarnation {0}]', incarnation)
+
+        vm_settings, vm_settings_updated = None, False
+        try:
+            vm_settings, vm_settings_updated = GoalState._fetch_vm_settings(self._wire_client)
+        except VmSettingsSupportStopped as exception:  # If the HGAP stopped supporting vmSettings, we need to use the goal state from the WireServer
+            self._restore_wire_server_goal_state(incarnation, xml_text, xml_doc, exception)
+            return
 
-        vm_settings, vm_settings_updated = GoalState._fetch_vm_settings(self._wire_client)
         if vm_settings_updated:
             logger.info("Fetched new vmSettings [HostGAPlugin correlation ID: {0} eTag: {1} source: {2}]", vm_settings.hostga_plugin_correlation_id, vm_settings.etag, vm_settings.source)
         # Ignore the vmSettings if their source is Fabric (processing a Fabric goal state may require the tenant certificate and the vmSettings don't include it.)
@@ -177,6 +184,18 @@ class GoalState(object):
         if self._extensions_goal_state is None or most_recent.created_on_timestamp > self._extensions_goal_state.created_on_timestamp:
             self._extensions_goal_state = most_recent
 
+    def _restore_wire_server_goal_state(self, incarnation, xml_text, xml_doc, vm_settings_support_stopped_error):
+        logger.info('The HGAP stopped supporting vmSettings; will fetched the goal state from the WireServer.')
+        self._history = GoalStateHistory(timeutil.create_timestamp(), incarnation)
+        self._history.save_goal_state(xml_text)
+        self._extensions_goal_state = self._fetch_full_wire_server_goal_state(incarnation, xml_doc)
+        if self._extensions_goal_state.created_on_timestamp < vm_settings_support_stopped_error.timestamp:
+            self._extensions_goal_state.is_outdated = True
+            msg = "Fetched a Fabric goal state older than the most recent FastTrack goal state; will skip it. (Fabric: {0} FastTrack: {1})".format(
+                  self._extensions_goal_state.created_on_timestamp, vm_settings_support_stopped_error.timestamp)
+            logger.info(msg)
+            add_event(op=WALAEventOperation.VmSettings, message=msg, is_success=True)
+
     def save_to_history(self, data, file_name):
         self._history.save(data, file_name)
 
@@ -231,6 +250,8 @@ class GoalState(object):
                     GoalState.update_host_plugin_headers(wire_client)
                     vm_settings, vm_settings_updated = wire_client.get_host_plugin().fetch_vm_settings()
 
+            except VmSettingsSupportStopped:
+                raise
             except VmSettingsNotSupported:
                 pass
             except VmSettingsParseError as exception:
@@ -249,7 +270,7 @@ class GoalState(object):
         Returns the value of ExtensionsConfig.
         """
         try:
-            logger.info('Fetching full goal state from the WireServer')
+            logger.info('Fetching full goal state from the WireServer [incarnation {0}]', incarnation)
 
             role_instance = find(xml_doc, "RoleInstance")
             role_instance_id = findtext(role_instance, "InstanceId")
diff --git a/azurelinuxagent/common/protocol/hostplugin.py b/azurelinuxagent/common/protocol/hostplugin.py
index 2e895ff6..477ccf9b 100644
--- a/azurelinuxagent/common/protocol/hostplugin.py
+++ b/azurelinuxagent/common/protocol/hostplugin.py
@@ -16,23 +16,22 @@
 #
 # Requires Python 2.6+ and Openssl 1.0+
 #
-
 import base64
 import datetime
 import json
+import os.path
 import uuid
 
-from azurelinuxagent.common import logger
+from azurelinuxagent.common import logger, conf
 from azurelinuxagent.common.errorstate import ErrorState, ERROR_STATE_HOST_PLUGIN_FAILURE
 from azurelinuxagent.common.event import WALAEventOperation, add_event
 from azurelinuxagent.common.exception import HttpError, ProtocolError, ResourceGoneError
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 from azurelinuxagent.common.future import ustr, httpclient
 from azurelinuxagent.common.protocol.healthservice import HealthService
-from azurelinuxagent.common.protocol.extensions_goal_state import VmSettingsParseError
+from azurelinuxagent.common.protocol.extensions_goal_state import VmSettingsParseError, GoalStateSource
 from azurelinuxagent.common.protocol.extensions_goal_state_factory import ExtensionsGoalStateFactory
-from azurelinuxagent.common.utils import restutil
-from azurelinuxagent.common.utils import textutil
+from azurelinuxagent.common.utils import restutil, textutil, timeutil
 from azurelinuxagent.common.utils.textutil import remove_bom
 from azurelinuxagent.common.version import AGENT_NAME, AGENT_VERSION, PY_VERSION_MAJOR
 
@@ -86,11 +85,21 @@ class HostPluginProtocol(object):
         self.status_error_state = ErrorState(min_timedelta=ERROR_STATE_HOST_PLUGIN_FAILURE)
         self.fetch_last_timestamp = None
         self.status_last_timestamp = None
-        self._host_plugin_version = FlexibleVersion("0.0.0.0")  # Version 0 means "unknown"
-        self._host_plugin_supports_vm_settings = None   # Tri-state variable: None == Not Initialized, True == Supports, False == Does Not Support
-        self._host_plugin_supports_vm_settings_next_check = datetime.datetime.now()
+        self._version = FlexibleVersion("0.0.0.0")  # Version 0 means "unknown"
+        self._supports_vm_settings = None   # Tri-state variable: None == Not Initialized, True == Supports, False == Does Not Support
+        self._supports_vm_settings_next_check = datetime.datetime.now()
         self._vm_settings_error_reporter = _VmSettingsErrorReporter()
-        self._cached_vm_settings = None  # Cached value of the most recent ExtensionsGoalStateFromVmSettings
+        self._cached_vm_settings = None  # Cached value of the most recent vmSettings
+
+        # restore the state of Fast Track
+        if not os.path.exists(self._get_fast_track_state_file()):
+            self._supports_vm_settings = False
+            self._supports_vm_settings_next_check = datetime.datetime.now()
+            self._fast_track_timestamp = None
+        else:
+            self._supports_vm_settings = True
+            self._supports_vm_settings_next_check = datetime.datetime.now()
+            self._fast_track_timestamp = HostPluginProtocol._get_fast_track_timestamp()
 
     @staticmethod
     def _extract_deployment_id(role_config_name):
@@ -102,9 +111,9 @@ class HostPluginProtocol(object):
         Returns True if the HostGAPlugin supports the vmSettings API.
         """
         # _host_plugin_supports_vm_settings is set by fetch_vm_settings()
-        if self._host_plugin_supports_vm_settings is None:
+        if self._supports_vm_settings is None:
             _, _ = self.fetch_vm_settings()
-        return self._host_plugin_supports_vm_settings
+        return self._supports_vm_settings
 
     def update_container_id(self, new_container_id):
         self.container_id = new_container_id
@@ -405,34 +414,75 @@ class HostPluginProtocol(object):
             return s.decode('utf-8')
         return s
 
+    @staticmethod
+    def _get_fast_track_state_file():
+        # This file keeps the timestamp of the most recent goal state if it was retrieved via Fast Track
+        return os.path.join(conf.get_lib_dir(), "fast_track.json")
+
+    @staticmethod
+    def _save_fast_track_state(timestamp):
+        try:
+            with open(HostPluginProtocol._get_fast_track_state_file(), "w") as file_:
+                json.dump({"timestamp": timestamp}, file_)
+        except Exception as e:
+            logger.warn("Error updating the Fast Track state ({0}): {1}", HostPluginProtocol._get_fast_track_state_file(), ustr(e))
+
+    @staticmethod
+    def clear_fast_track_state():
+        try:
+            if os.path.exists(HostPluginProtocol._get_fast_track_state_file()):
+                os.remove(HostPluginProtocol._get_fast_track_state_file())
+        except Exception as e:
+            logger.warn("Error clearing the current state for Fast Track ({0}): {1}", HostPluginProtocol._get_fast_track_state_file(),
+                        ustr(e))
+
+    @staticmethod
+    def _get_fast_track_timestamp():
+        try:
+            with open(HostPluginProtocol._get_fast_track_state_file(), "r") as file_:
+                return json.load(file_)["timestamp"]
+        except Exception as e:
+            logger.warn("Can't retrieve the timestamp for the most recent Fast Track goal state ({0}), will assume the current time. Error: {1}",
+                HostPluginProtocol._get_fast_track_state_file(), ustr(e))
+        return timeutil.create_timestamp(datetime.datetime.utcnow())
+
     def fetch_vm_settings(self):
         """
-        Queries the vmSettings from the HostGAPlugin and returns an (ExtensionsGoalStateFromVmSettings, bool) tuple with the vmSettings and
+        Queries the vmSettings from the HostGAPlugin and returns an (ExtensionsGoalState, bool) tuple with the vmSettings and
         a boolean indicating if they are an updated (True) or a cached value (False).
 
         Raises
             * VmSettingsNotSupported if the HostGAPlugin does not support the vmSettings API
-            * VmSettingsError if the HostGAPlugin returned invalid vmSettings (e.g. syntax error)
+            * VmSettingsSupportStopped if the HostGAPlugin stopped supporting the vmSettings API
+            * VmSettingsParseError if the HostGAPlugin returned invalid vmSettings (e.g. syntax error)
             * ResourceGoneError if the container ID and roleconfig name need to be refreshed
             * ProtocolError if the request fails for any other reason (e.g. not supported, time out, server error)
         """
-        def raise_not_supported(reset_state=False):
-            if reset_state:
-                self._host_plugin_supports_vm_settings = False
-                self._host_plugin_supports_vm_settings_next_check = datetime.datetime.now() + datetime.timedelta(hours=6)  # check again in 6 hours
-                # "Not supported" is not considered an error, so don't use self._vm_settings_error_reporter to report it
-                logger.info("vmSettings is not supported")
-                add_event(op=WALAEventOperation.HostPlugin, message="vmSettings is not supported", is_success=True)
-            raise VmSettingsNotSupported()
+        def raise_not_supported():
+            try:
+                if self._supports_vm_settings:
+                    # The most recent goal state was delivered using FastTrack, and suddenly the HostGAPlugin does not support the vmSettings API anymore.
+                    # This can happen if, for example, the VM is migrated across host nodes that are running different versions of the HostGAPlugin.
+                    logger.warn("The HostGAPlugin stopped supporting the vmSettings API. If there is a pending FastTrack goal state, it will not be executed.")
+                    add_event(op=WALAEventOperation.VmSettings, message="[VmSettingsSupportStopped] HostGAPlugin: {0}".format(self._version), is_success=False, log_event=False)
+                    raise VmSettingsSupportStopped(self._fast_track_timestamp)
+                else:
+                    logger.info("HostGAPlugin {0} does not support the vmSettings API. Will not use FastTrack.", self._version)
+                    add_event(op=WALAEventOperation.VmSettings, message="[VmSettingsNotSupported] HostGAPlugin: {0}".format(self._version), is_success=True)
+                    raise VmSettingsNotSupported()
+            finally:
+                self._supports_vm_settings = False
+                self._supports_vm_settings_next_check = datetime.datetime.now() + datetime.timedelta(hours=6)  # check again in 6 hours
 
         def format_message(msg):
             return "GET vmSettings [correlation ID: {0} eTag: {1}]: {2}".format(correlation_id, etag, msg)
 
         try:
-            # Raise if VmSettings are not supported but check for periodically since the HostGAPlugin could have been updated since the last check
+            # Raise if VmSettings are not supported, but check again periodically since the HostGAPlugin could have been updated since the last check
             # Note that self._host_plugin_supports_vm_settings can be None, so we need to compare against False
-            if self._host_plugin_supports_vm_settings == False and self._host_plugin_supports_vm_settings_next_check > datetime.datetime.now():
-                raise_not_supported()
+            if self._supports_vm_settings == False and self._supports_vm_settings_next_check > datetime.datetime.now():
+                # Raise VmSettingsNotSupported directly instead of using raise_not_supported() to avoid resetting the timestamp for the next check
+                raise VmSettingsNotSupported()
 
             etag = None if self._cached_vm_settings is None else self._cached_vm_settings.etag
             correlation_id = str(uuid.uuid4())
@@ -448,7 +498,7 @@ class HostPluginProtocol(object):
                 raise ResourceGoneError()
 
             if response.status == httpclient.NOT_FOUND:  # the HostGAPlugin does not support FastTrack
-                raise_not_supported(reset_state=True)
+                raise_not_supported()
 
             if response.status == httpclient.NOT_MODIFIED:  # The goal state hasn't changed, return the current instance
                 return self._cached_vm_settings, False
@@ -467,7 +517,7 @@ class HostPluginProtocol(object):
                 elif 500 <= response.status <= 599:
                     self._vm_settings_error_reporter.report_error(error_description, _VmSettingsError.ServerError)
                 else:
-                    self._vm_settings_error_reporter.report_error(error_description)
+                    self._vm_settings_error_reporter.report_error(error_description, _VmSettingsError.HttpError)
 
                 raise ProtocolError(error_description)
 
@@ -477,7 +527,6 @@ class HostPluginProtocol(object):
                     break
             else:  # since the vmSettings were updated, the response must include an etag
                 message = format_message("The vmSettings response does not include an Etag header")
-                self._vm_settings_error_reporter.report_error(message)
                 raise ProtocolError(message)
 
             response_content = ustr(response.read(), encoding='utf-8')
@@ -485,8 +534,8 @@ class HostPluginProtocol(object):
             vm_settings = ExtensionsGoalStateFactory.create_from_vm_settings(response_etag, response_content, correlation_id)
 
             # log the HostGAPlugin version
-            if vm_settings.host_ga_plugin_version != self._host_plugin_version:
-                self._host_plugin_version = vm_settings.host_ga_plugin_version
+            if vm_settings.host_ga_plugin_version != self._version:
+                self._version = vm_settings.host_ga_plugin_version
                 message = "HostGAPlugin version: {0}".format(vm_settings.host_ga_plugin_version)
                 logger.info(message)
                 add_event(op=WALAEventOperation.HostPlugin, message=message, is_success=True)
@@ -494,17 +543,20 @@ class HostPluginProtocol(object):
             # Don't support HostGAPlugin versions older than 123
             # TODO: update the minimum version to 1.0.8.123 before release
             if vm_settings.host_ga_plugin_version < FlexibleVersion("1.0.8.117"):
-                raise_not_supported(reset_state=True)
+                raise_not_supported()
 
-            self._host_plugin_supports_vm_settings = True
+            self._supports_vm_settings = True
             self._cached_vm_settings = vm_settings
+
+            if vm_settings.source == GoalStateSource.FastTrack:
+                self._fast_track_timestamp = vm_settings.created_on_timestamp
+                self._save_fast_track_state(vm_settings.created_on_timestamp)
+            else:
+                self.clear_fast_track_state()
+
             return vm_settings, True
 
-        except (ProtocolError, ResourceGoneError, VmSettingsNotSupported):
-            raise
-        except VmSettingsParseError as exception:
-            message = format_message(ustr(exception))
-            self._vm_settings_error_reporter.report_error(message)
+        except (ProtocolError, ResourceGoneError, VmSettingsNotSupported, VmSettingsParseError):
             raise
         except Exception as exception:
             if isinstance(exception, IOError) and "timed out" in ustr(exception):
@@ -519,19 +571,31 @@ class HostPluginProtocol(object):
 
 
 class VmSettingsNotSupported(TypeError):
-    pass
+    """
+    Indicates that the HostGAPlugin does not support the vmSettings API
+    """
+
+
+class VmSettingsSupportStopped(VmSettingsNotSupported):
+    """
+    Indicates that the HostGAPlugin supported the vmSettings API in previous calls, but now it does not support it for current call.
+    This can happen, for example, if the VM is migrated across nodes with different HostGAPlugin versions.
+    """
+    def __init__(self, timestamp):
+        super(VmSettingsSupportStopped, self).__init__()
+        self.timestamp = timestamp
 
 
 class _VmSettingsError(object):
-    ServerError   = 'ServerError'
-    ClientError   = 'ClientError'
-    Timeout       = 'Timeout'
-    RequestFailed = 'RequestFailed'
+    ClientError    = 'ClientError'
+    HttpError      = 'HttpError'
+    RequestFailed  = 'RequestFailed'
+    ServerError    = 'ServerError'
+    Timeout        = 'Timeout'
 
 
 class _VmSettingsErrorReporter(object):
-    _MaxLogErrors = 1  # Max number of errors by period reported to the local log
-    _MaxTelemetryErrors = 3  # Max number of errors  by period reported to telemetry
+    _MaxErrors = 3  # Max number of errors reported to telemetry (by period)
     _Period = datetime.timedelta(hours=1)  # How often to report the summary
 
     def __init__(self):
@@ -540,32 +604,32 @@ class _VmSettingsErrorReporter(object):
     def _reset(self):
         self._request_count = 0  # Total number of vmSettings HTTP requests
         self._error_count = 0   # Total number of errors issuing vmSettings requests (includes all kinds of errors)
-        self._server_error_count = 0  # Count of server side errors (HTTP status in the 500s)
         self._client_error_count = 0  # Count of client side errors (HTTP status in the 400s)
-        self._timeout_count = 0  # Count of timeouts on vmSettings requests
+        self._http_error_count = 0  # Count of HTTP errors other than 400s and 500s
         self._request_failure_count = 0  # Total count of requests that could not be issued (does not include timeouts or requests that were actually issued and failed, for example, with 500 or 400 statuses)
+        self._server_error_count = 0  # Count of server side errors (HTTP status in the 500s)
+        self._timeout_count = 0  # Count of timeouts on vmSettings requests
         self._next_period = datetime.datetime.now() + _VmSettingsErrorReporter._Period
 
     def report_request(self):
         self._request_count += 1
 
-    def report_error(self, error, category=None):
+    def report_error(self, error, category):
         self._error_count += 1
 
-        if self._error_count <= _VmSettingsErrorReporter._MaxLogErrors:
-            logger.info("[VmSettings] [Informational only, the Agent will continue normal operation] {0}", error)
-
-        if self._error_count <= _VmSettingsErrorReporter._MaxTelemetryErrors:
-            add_event(op=WALAEventOperation.VmSettings, message=error, is_success=False, log_event=False)
+        if self._error_count <= _VmSettingsErrorReporter._MaxErrors:
+            add_event(op=WALAEventOperation.VmSettings, message="[{0}] {1}".format(category, error), is_success=False, log_event=False)
 
-        if category == _VmSettingsError.ServerError:
-            self._server_error_count += 1
-        elif category == _VmSettingsError.ClientError:
+        if category == _VmSettingsError.ClientError:
             self._client_error_count += 1
-        elif category == _VmSettingsError.Timeout:
-            self._timeout_count += 1
+        elif category == _VmSettingsError.HttpError:
+            self._http_error_count += 1
         elif category == _VmSettingsError.RequestFailed:
             self._request_failure_count += 1
+        elif category == _VmSettingsError.ServerError:
+            self._server_error_count += 1
+        elif category == _VmSettingsError.Timeout:
+            self._timeout_count += 1
 
     def report_summary(self):
         if datetime.datetime.now() >= self._next_period:
diff --git a/azurelinuxagent/ga/update.py b/azurelinuxagent/ga/update.py
index 582716ac..7e8f08b6 100644
--- a/azurelinuxagent/ga/update.py
+++ b/azurelinuxagent/ga/update.py
@@ -567,7 +567,8 @@ class UpdateHandler(object):
         """
         True if we are currently processing a new extensions goal state
         """
-        return self._goal_state is not None and self._goal_state.extensions_goal_state.id != self._last_extensions_gs_id
+        egs = self._goal_state.extensions_goal_state
+        return self._goal_state is not None and egs.id != self._last_extensions_gs_id and not egs.is_outdated
 
     def _process_goal_state(self, exthandlers_handler, remote_access_handler):
         try:
@@ -670,6 +671,8 @@ class UpdateHandler(object):
             if self._goal_state is not None:
                 agent_status = exthandlers_handler.get_ext_handlers_status_debug_info(vm_status)
                 self._goal_state.save_to_history(agent_status, AGENT_STATUS_FILE)
+                if self._goal_state.extensions_goal_state.is_outdated:
+                    exthandlers_handler.protocol.client.get_host_plugin().clear_fast_track_state()
 
     def _report_extensions_summary(self, vm_status):
         try:


Test patch:
diff --git a/tests/ga/test_update.py b/tests/ga/test_update.py
index 22594056..d9e1fe04 100644
--- a/tests/ga/test_update.py
+++ b/tests/ga/test_update.py
@@ -1356,26 +1356,27 @@ class TestUpdate(UpdateTestCase):
                                     with patch('azurelinuxagent.ga.update.get_collect_telemetry_events_handler') as mock_event_collector:
                                         with patch('azurelinuxagent.ga.update.initialize_event_logger_vminfo_common_parameters'):
                                             with patch('azurelinuxagent.ga.update.is_log_collection_allowed', return_value=True):
-                                                with patch('time.sleep') as sleep_mock:
-                                                    with patch('sys.exit') as mock_exit:
-                                                        if isinstance(os.getppid, MagicMock):
-                                                            self.update_handler.run()
-                                                        else:
-                                                            with patch('os.getppid', return_value=42):
+                                                with patch.object(self.update_handler, "_processing_new_extensions_goal_state", return_value=True):
+                                                    with patch('time.sleep') as sleep_mock:
+                                                        with patch('sys.exit') as mock_exit:
+                                                            if isinstance(os.getppid, MagicMock):
                                                                 self.update_handler.run()
-
-                                                    self.assertEqual(1, mock_handler.call_count)
-                                                    self.assertEqual(calls, len([c for c in [call[0] for call in mock_handler.return_value.method_calls] if c == 'run']))
-                                                    self.assertEqual(1, mock_ra_handler.call_count)
-                                                    self.assertEqual(calls, len(mock_ra_handler.return_value.method_calls))
-                                                    if calls > 0:
-                                                        self.assertEqual(sleep_interval, sleep_mock.call_args[0])
-                                                    self.assertEqual(1, mock_monitor.call_count)
-                                                    self.assertEqual(1, mock_env.call_count)
-                                                    self.assertEqual(1, mock_collect_logs.call_count)
-                                                    self.assertEqual(1, mock_telemetry_send_events.call_count)
-                                                    self.assertEqual(1, mock_event_collector.call_count)
-                                                    self.assertEqual(1, mock_exit.call_count)
+                                                            else:
+                                                                with patch('os.getppid', return_value=42):
+                                                                    self.update_handler.run()
+
+                                                        self.assertEqual(1, mock_handler.call_count)
+                                                        self.assertEqual(calls, len([c for c in [call[0] for call in mock_handler.return_value.method_calls] if c == 'run']))
+                                                        self.assertEqual(1, mock_ra_handler.call_count)
+                                                        self.assertEqual(calls, len(mock_ra_handler.return_value.method_calls))
+                                                        if calls > 0:
+                                                            self.assertEqual(sleep_interval, sleep_mock.call_args[0])
+                                                        self.assertEqual(1, mock_monitor.call_count)
+                                                        self.assertEqual(1, mock_env.call_count)
+                                                        self.assertEqual(1, mock_collect_logs.call_count)
+                                                        self.assertEqual(1, mock_telemetry_send_events.call_count)
+                                                        self.assertEqual(1, mock_event_collector.call_count)
+                                                        self.assertEqual(1, mock_exit.call_count)
 
     def test_run(self):
         self._test_run()
@@ -1463,6 +1464,7 @@ class TestUpdate(UpdateTestCase):
 
         self.update_handler.protocol_util = protocol
         self.update_handler._goal_state = protocol.get_goal_state()
+        self.update_handler._goal_state.extensions_goal_state.is_outdated = False
         conf.get_autoupdate_gafamily = Mock(return_value=protocol.family)
 
         return self.update_handler._download_agent_if_upgrade_available(protocol, base_version=base_version)
diff --git a/tests/protocol/test_goal_state.py b/tests/protocol/test_goal_state.py
index ce1c2964..65d050d8 100644
--- a/tests/protocol/test_goal_state.py
+++ b/tests/protocol/test_goal_state.py
@@ -10,6 +10,9 @@ import time
 
 from azurelinuxagent.common.future import httpclient
 from azurelinuxagent.common.protocol.extensions_goal_state import GoalStateSource, GoalStateChannel
+from azurelinuxagent.common.protocol.extensions_goal_state_from_extensions_config import ExtensionsGoalStateFromExtensionsConfig
+from azurelinuxagent.common.protocol.extensions_goal_state_from_vm_settings import ExtensionsGoalStateFromVmSettings
+from azurelinuxagent.common.protocol import hostplugin
 from azurelinuxagent.common.protocol.goal_state import GoalState, _GET_GOAL_STATE_MAX_ATTEMPTS
 from azurelinuxagent.common.exception import ProtocolError
 from azurelinuxagent.common.utils import fileutil
@@ -20,7 +23,68 @@ from tests.protocol.HttpRequestPredicates import HttpRequestPredicates
 from tests.tools import AgentTestCase, patch, load_data
 
 
-class GoalStateTestCase(AgentTestCase):
+class GoalStateTestCase(AgentTestCase, HttpRequestPredicates):
+    def test_it_should_use_vm_settings_by_default(self):
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+            extensions_goal_state = GoalState(protocol.client).extensions_goal_state
+            self.assertTrue(
+                isinstance(extensions_goal_state, ExtensionsGoalStateFromVmSettings),
+                'The extensions goal state should have been created from the vmSettings (got: {0})'.format(type(extensions_goal_state)))
+
+    def _assert_is_extensions_goal_state_from_extensions_config(self, extensions_goal_state):
+        self.assertTrue(
+            isinstance(extensions_goal_state, ExtensionsGoalStateFromExtensionsConfig),
+            'The extensions goal state should have been created from the extensionsConfig (got: {0})'.format(type(extensions_goal_state)))
+
+    def test_it_should_use_extensions_config_when_fast_track_is_disabled(self):
+        with patch("azurelinuxagent.common.conf.get_enable_fast_track", return_value=False):
+            with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+                self._assert_is_extensions_goal_state_from_extensions_config(GoalState(protocol.client).extensions_goal_state)
+
+    def test_it_should_use_extensions_config_when_fast_track_is_not_supported(self):
+        def http_get_handler(url, *_, **__):
+            if self.is_host_plugin_vm_settings_request(url):
+                return MockHttpResponse(httpclient.NOT_FOUND)
+            return None
+
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS, http_get_handler=http_get_handler) as protocol:
+            self._assert_is_extensions_goal_state_from_extensions_config(GoalState(protocol.client).extensions_goal_state)
+
+    def test_it_should_use_extensions_config_when_the_host_ga_plugin_version_is_not_supported(self):
+        data_file = mockwiredata.DATA_FILE_VM_SETTINGS.copy()
+        data_file["vm_settings"] = "hostgaplugin/vm_settings-unsupported_version.json"
+
+        with mock_wire_protocol(data_file) as protocol:
+            self._assert_is_extensions_goal_state_from_extensions_config(GoalState(protocol.client).extensions_goal_state)
+
+    def test_it_should_retry_get_vm_settings_on_resource_gone_error(self):
+        # Requests to the hostgaplugin incude the Container ID and the RoleConfigName as headers; when the hostgaplugin returns GONE (HTTP status 410) the agent
+        # needs to get a new goal state and retry the request with updated values for the Container ID and RoleConfigName headers.
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+            # Do not mock the vmSettings request at the level of azurelinuxagent.common.utils.restutil.http_request. The GONE status is handled
+            # in the internal _http_request, which we mock below.
+            protocol.do_not_mock = lambda method, url: method == "GET" and self.is_host_plugin_vm_settings_request(url)
+
+            request_headers = []  # we expect a retry with new headers and use this array to persist the headers of each request
+
+            def http_get_vm_settings(_method, _host, _relative_url, **kwargs):
+                request_headers.append(kwargs["headers"])
+                if len(request_headers) == 1:
+                    # Fail the first request with status GONE and update the mock data to return the new Container ID and RoleConfigName that should be
+                    # used in the headers of the retry request.
+                    protocol.mock_wire_data.set_container_id("GET_VM_SETTINGS_TEST_CONTAINER_ID")
+                    protocol.mock_wire_data.set_role_config_name("GET_VM_SETTINGS_TEST_ROLE_CONFIG_NAME")
+                    return MockHttpResponse(status=httpclient.GONE)
+                # For this test we are interested only on the retry logic, so the second request (the retry) is not important; we use NOT_MODIFIED (304) for simplicity.
+                return MockHttpResponse(status=httpclient.NOT_MODIFIED)
+
+            with patch("azurelinuxagent.common.utils.restutil._http_request", side_effect=http_get_vm_settings):
+                protocol.client.update_goal_state()
+
+            self.assertEqual(2, len(request_headers), "We expected 2 requests for vmSettings: the original request and the retry request")
+            self.assertEqual("GET_VM_SETTINGS_TEST_CONTAINER_ID", request_headers[1][hostplugin._HEADER_CONTAINER_ID], "The retry request did not include the expected header for the ContainerId")
+            self.assertEqual("GET_VM_SETTINGS_TEST_ROLE_CONFIG_NAME", request_headers[1][hostplugin._HEADER_HOST_CONFIG_NAME], "The retry request did not include the expected header for the RoleConfigName")
+
     def test_fetch_goal_state_should_raise_on_incomplete_goal_state(self):
         with mock_wire_protocol(mockwiredata.DATA_FILE) as protocol:
             protocol.mock_wire_data.data_files = mockwiredata.DATA_FILE_NOOP_GS
@@ -166,13 +230,13 @@ class GoalStateTestCase(AgentTestCase):
             # that may invalidate this setup.
             vm_settings, _ = protocol.client.get_host_plugin().fetch_vm_settings()
             if vm_settings.etag != etag:
-                raise Exception("The HostGAPlugin is no in sync. Expected ETag {0}. Got {1}".format(etag, vm_settings.etag))
+                raise Exception("The HostGAPlugin is not in sync. Expected ETag {0}. Got {1}".format(etag, vm_settings.etag))
             if vm_settings.source != GoalStateSource.Fabric:
                 raise Exception("The HostGAPlugin should be returning a Fabric goal state. Got {0}".format(vm_settings.source))
 
             goal_state = GoalState(protocol.client)
             if goal_state.incarnation != incarnation:
-                raise Exception("The WireServer is no in sync. Expected incarnation {0}. Got {1}".format(incarnation, goal_state.incarnation))
+                raise Exception("The WireServer is not in sync. Expected incarnation {0}. Got {1}".format(incarnation, goal_state.incarnation))
 
             if goal_state.extensions_goal_state.correlation_id != vm_settings.correlation_id:
                 raise Exception(
@@ -267,3 +331,30 @@ class GoalStateTestCase(AgentTestCase):
             goal_state.update()
 
             self._assert_goal_state(goal_state, '222', channel=GoalStateChannel.WireServer, source=GoalStateSource.Fabric)
+
+    def test_it_should_mark_outdated_goal_states(self):
+        with GoalStateTestCase._create_protocol_ws_and_hgap_in_sync() as protocol:
+            goal_state = GoalState(protocol.client)
+            initial_incarnation = goal_state.incarnation
+            initial_timestamp = goal_state.extensions_goal_state.created_on_timestamp
+
+            # Make the most recent goal state FastTrack
+            timestamp = datetime.datetime.utcnow() + datetime.timedelta(seconds=15)
+            protocol.mock_wire_data.set_vm_settings_source(GoalStateSource.FastTrack)
+            protocol.mock_wire_data.set_etag('444444', timestamp)
+
+            goal_state.update()
+
+            # Update the goal state after the HGAP plugin stops supporting vmSettings
+            def http_get_handler(url, *_, **__):
+                if self.is_host_plugin_vm_settings_request(url):
+                    return MockHttpResponse(httpclient.NOT_FOUND)
+                return None
+
+            protocol.set_http_handlers(http_get_handler=http_get_handler)
+
+            goal_state.update()
+
+            self._assert_goal_state(goal_state, initial_incarnation, channel=GoalStateChannel.WireServer, source=GoalStateSource.Fabric)
+            self.assertEqual(initial_timestamp, goal_state.extensions_goal_state.created_on_timestamp, "The timestamp of the updated goal state is incorrect")
+            self.assertTrue(goal_state.extensions_goal_state.is_outdated, "The updated goal state should be marked as outdated")
diff --git a/tests/protocol/test_hostplugin.py b/tests/protocol/test_hostplugin.py
index f3b5a588..16bb7ef0 100644
--- a/tests/protocol/test_hostplugin.py
+++ b/tests/protocol/test_hostplugin.py
@@ -19,17 +19,20 @@ import base64
 import contextlib
 import datetime
 import json
+import os.path
 import sys
 import unittest
 
 import azurelinuxagent.common.protocol.hostplugin as hostplugin
 import azurelinuxagent.common.protocol.restapi as restapi
 import azurelinuxagent.common.protocol.wire as wire
+from azurelinuxagent.common import conf
 from azurelinuxagent.common.errorstate import ErrorState
 from azurelinuxagent.common.exception import HttpError, ResourceGoneError, ProtocolError
 from azurelinuxagent.common.future import ustr, httpclient
 from azurelinuxagent.common.osutil.default import UUID_PATTERN
-from azurelinuxagent.common.protocol.hostplugin import API_VERSION, _VmSettingsErrorReporter, VmSettingsNotSupported
+from azurelinuxagent.common.protocol.hostplugin import API_VERSION, _VmSettingsErrorReporter, VmSettingsNotSupported, VmSettingsSupportStopped
+from azurelinuxagent.common.protocol.extensions_goal_state import GoalStateSource
 from azurelinuxagent.common.protocol.goal_state import GoalState
 from azurelinuxagent.common.utils import restutil
 from azurelinuxagent.common.version import AGENT_VERSION, AGENT_NAME
@@ -926,40 +929,23 @@ class TestHostPluginVmSettings(HttpRequestPredicates, AgentTestCase):
             def get_telemetry_messages():
                 return [kwargs["message"] for _, kwargs in add_event.call_args_list if kwargs["op"] == "VmSettings"]
 
-            def get_log_messages():
-                return [arg[0][0] for arg in logger_info.call_args_list   if "[VmSettings]" in arg[0][0]]
-
-            def fetch_vm_settings():
-                try:
-                    host_plugin.fetch_vm_settings()
-                except ProtocolError:
-                    pass  # All calls produce an error; ignore it
-
             with patch("azurelinuxagent.common.protocol.hostplugin.add_event") as add_event:
-                with patch('azurelinuxagent.common.logger.info') as logger_info:
-                    host_plugin = protocol.client.get_host_plugin()
-                    for _ in range(_VmSettingsErrorReporter._MaxTelemetryErrors + 3):
-                        fetch_vm_settings()
+                for _ in range(_VmSettingsErrorReporter._MaxErrors + 3):
+                    self._fetch_vm_settings_ignoring_errors(protocol)
 
-                    telemetry_messages = get_telemetry_messages()
-                    self.assertEqual(_VmSettingsErrorReporter._MaxTelemetryErrors, len(telemetry_messages), "The number of errors reported to telemetry is not the max allowed (got: {0})".format(telemetry_messages))
-
-                    log_messages = get_log_messages()
-                    self.assertEqual(_VmSettingsErrorReporter._MaxLogErrors, len(log_messages), "The number of errors reported to the local log is not the max allowed (got: {0})".format(telemetry_messages))
+                telemetry_messages = get_telemetry_messages()
+                self.assertEqual(_VmSettingsErrorReporter._MaxErrors, len(telemetry_messages), "The number of errors reported to telemetry is not the max allowed (got: {0})".format(telemetry_messages))
 
             # Reset the error reporter and verify that additional errors are reported
-            host_plugin._vm_settings_error_reporter._next_period = datetime.datetime.now()
-            fetch_vm_settings()  # this triggers the reset
+            protocol.client.get_host_plugin()._vm_settings_error_reporter._next_period = datetime.datetime.now()
+            self._fetch_vm_settings_ignoring_errors(protocol)  # this triggers the reset
 
             with patch("azurelinuxagent.common.protocol.hostplugin.add_event") as add_event:
-                fetch_vm_settings()
+                self._fetch_vm_settings_ignoring_errors(protocol)
 
                 telemetry_messages = get_telemetry_messages()
                 self.assertEqual(1, len(telemetry_messages), "Expected additional errors to be reported to telemetry in the next period (got: {0})".format(telemetry_messages))
 
-                log_messages = get_log_messages()
-                self.assertEqual(1, len(log_messages), "Expected additional errors to be reported to the local log in the next period (got: {0})".format(telemetry_messages))
-
     def test_it_should_stop_issuing_vm_settings_requests_when_api_is_not_supported(self):
         with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
             def http_get_handler(url, *_, **__):
@@ -974,15 +960,54 @@ class TestHostPluginVmSettings(HttpRequestPredicates, AgentTestCase):
             self._fetch_vm_settings_ignoring_errors(protocol)
             self.assertEqual(1, get_vm_settings_call_count(), "There should have been an initial call to vmSettings.")
 
-            protocol.client.update_goal_state()
-            protocol.client.update_goal_state()
+            self._fetch_vm_settings_ignoring_errors(protocol)
+            self._fetch_vm_settings_ignoring_errors(protocol)
             self.assertEqual(1, get_vm_settings_call_count(), "Additional calls to update_goal_state should not have produced extra calls to vmSettings.")
 
             # reset the vmSettings check period; this should restart the calls to the API
-            protocol.client._host_plugin._host_plugin_supports_vm_settings_next_check = datetime.datetime.now()
+            protocol.client._host_plugin._supports_vm_settings_next_check = datetime.datetime.now()
             protocol.client.update_goal_state()
             self.assertEqual(2, get_vm_settings_call_count(), "A second call to vmSettings was expecting after the check period has elapsed.")
 
+    def test_it_should_raise_when_the_vm_settings_api_stops_being_supported(self):
+        def http_get_handler(url, *_, **__):
+            if self.is_host_plugin_vm_settings_request(url):
+                return MockHttpResponse(httpclient.NOT_FOUND)  # HostGAPlugin returns 404 if the API is not supported
+            return None
+
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+            host_ga_plugin = protocol.client.get_host_plugin()
+
+            # Do an initial call to ensure the API is supported
+            vm_settings, _ = host_ga_plugin.fetch_vm_settings()
+
+            # Now return NOT_FOUND to indicate the API is not supported
+            protocol.set_http_handlers(http_get_handler=http_get_handler)
+
+            with self.assertRaises(VmSettingsSupportStopped) as cm:
+                host_ga_plugin.fetch_vm_settings()
+
+            self.assertEqual(vm_settings.created_on_timestamp, cm.exception.timestamp)
+
+    def test_it_should_save_the_timestamp_of_the_most_recent_fast_track_goal_state(self):
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+            host_ga_plugin = protocol.client.get_host_plugin()
+
+            vm_settings, _ = host_ga_plugin.fetch_vm_settings()
+
+            state_file = os.path.join(conf.get_lib_dir(), "fast_track.json")
+            self.assertTrue(os.path.exists(state_file), "The timestamp was not saved (can't find {0})".format(state_file))
+
+            with open(state_file, "r") as state_file_:
+                state = json.load(state_file_)
+            self.assertEqual(vm_settings.created_on_timestamp, state["timestamp"], "{0} does not contain the expected timestamp".format(state_file))
+
+            # A fabric goal state should remove the state file
+            protocol.mock_wire_data.set_vm_settings_source(GoalStateSource.Fabric)
+
+            _ = host_ga_plugin.fetch_vm_settings()
+
+            self.assertFalse(os.path.exists(state_file), "{0} was not removed by a Fabric goal state".format(state_file))
 
 class MockResponse:
     def __init__(self, body, status_code, reason=''):
diff --git a/tests/protocol/test_wire.py b/tests/protocol/test_wire.py
index 0b619c66..0cc8a01e 100644
--- a/tests/protocol/test_wire.py
+++ b/tests/protocol/test_wire.py
@@ -26,13 +26,10 @@ import uuid
 
 from azurelinuxagent.common.agent_supported_feature import SupportedFeatureNames, get_supported_feature_by_name, \
     get_agent_supported_features_list_for_crp
-from azurelinuxagent.common.future import httpclient
 from azurelinuxagent.common.event import WALAEventOperation
 from azurelinuxagent.common.exception import ResourceGoneError, ProtocolError, \
     ExtensionDownloadError, HttpError
-from azurelinuxagent.common.protocol import hostplugin
 from azurelinuxagent.common.protocol.extensions_goal_state_from_extensions_config import ExtensionsGoalStateFromExtensionsConfig
-from azurelinuxagent.common.protocol.extensions_goal_state_from_vm_settings import ExtensionsGoalStateFromVmSettings
 from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol
 from azurelinuxagent.common.protocol.wire import WireProtocol, WireClient, \
     StatusBlob, VMStatus
@@ -1097,67 +1094,6 @@ class UpdateGoalStateTestCase(HttpRequestPredicates, AgentTestCase):
             self.assertEqual(protocol.client.get_host_plugin().container_id, new_container_id)
             self.assertEqual(protocol.client.get_host_plugin().role_config_name, new_role_config_name)
 
-    def test_it_should_retry_get_vm_settings_on_resource_gone_error(self):
-        # Requests to the hostgaplugin incude the Container ID and the RoleConfigName as headers; when the hostgaplugin returns GONE (HTTP status 410) the agent
-        # needs to get a new goal state and retry the request with updated values for the Container ID and RoleConfigName headers.
-        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
-            # Do not mock the vmSettings request at the level of azurelinuxagent.common.utils.restutil.http_request. The GONE status is handled
-            # in the internal _http_request, which we mock below.
-            protocol.do_not_mock = lambda method, url: method == "GET" and self.is_host_plugin_vm_settings_request(url)
-
-            request_headers = []  # we expect a retry with new headers and use this array to persist the headers of each request
-
-            def http_get_vm_settings(_method, _host, _relative_url, **kwargs):
-                request_headers.append(kwargs["headers"])
-                if len(request_headers) == 1:
-                    # Fail the first request with status GONE and update the mock data to return the new Container ID and RoleConfigName that should be
-                    # used in the headers of the retry request.
-                    protocol.mock_wire_data.set_container_id("GET_VM_SETTINGS_TEST_CONTAINER_ID")
-                    protocol.mock_wire_data.set_role_config_name("GET_VM_SETTINGS_TEST_ROLE_CONFIG_NAME")
-                    return MockHttpResponse(status=httpclient.GONE)
-                # For this test we are interested only on the retry logic, so the second request (the retry) is not important; we use NOT_MODIFIED (304) for simplicity.
-                return MockHttpResponse(status=httpclient.NOT_MODIFIED)
-
-            with patch("azurelinuxagent.common.utils.restutil._http_request", side_effect=http_get_vm_settings):
-                protocol.client.update_goal_state()
-
-            self.assertEqual(2, len(request_headers), "We expected 2 requests for vmSettings: the original request and the retry request")
-            self.assertEqual("GET_VM_SETTINGS_TEST_CONTAINER_ID", request_headers[1][hostplugin._HEADER_CONTAINER_ID], "The retry request did not include the expected header for the ContainerId")
-            self.assertEqual("GET_VM_SETTINGS_TEST_ROLE_CONFIG_NAME", request_headers[1][hostplugin._HEADER_HOST_CONFIG_NAME], "The retry request did not include the expected header for the RoleConfigName")
-
-    def test_it_should_use_vm_settings_by_default(self):
-        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
-            extensions_goal_state = protocol.get_goal_state().extensions_goal_state
-            self.assertTrue(
-                isinstance(extensions_goal_state, ExtensionsGoalStateFromVmSettings),
-                'The extensions goal state should have been created from the vmSettings (got: {0})'.format(type(extensions_goal_state)))
-
-    def _assert_is_extensions_goal_state_from_extensions_config(self, extensions_goal_state):
-        self.assertTrue(
-            isinstance(extensions_goal_state, ExtensionsGoalStateFromExtensionsConfig),
-            'The extensions goal state should have been created from the extensionsConfig (got: {0})'.format(type(extensions_goal_state)))
-
-    def test_it_should_use_extensions_config_when_fast_track_is_disabled(self):
-        with patch("azurelinuxagent.common.conf.get_enable_fast_track", return_value=False):
-            with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
-                self._assert_is_extensions_goal_state_from_extensions_config(protocol.get_goal_state().extensions_goal_state)
-
-    def test_it_should_use_extensions_config_when_fast_track_is_not_supported(self):
-        def http_get_handler(url, *_, **__):
-            if self.is_host_plugin_vm_settings_request(url):
-                return MockHttpResponse(httpclient.NOT_FOUND)
-            return None
-
-        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS, http_get_handler=http_get_handler) as protocol:
-            self._assert_is_extensions_goal_state_from_extensions_config(protocol.get_goal_state().extensions_goal_state)
-
-    def test_it_should_use_extensions_config_when_the_host_ga_plugin_version_is_not_supported(self):
-        data_file = mockwiredata.DATA_FILE_VM_SETTINGS.copy()
-        data_file["vm_settings"] = "hostgaplugin/vm_settings-unsupported_version.json"
-
-        with mock_wire_protocol(data_file) as protocol:
-            self._assert_is_extensions_goal_state_from_extensions_config(protocol.get_goal_state().extensions_goal_state)
-
 
 class UpdateHostPluginFromGoalStateTestCase(AgentTestCase):
     """