Repository: Azure/WALinuxAgent
Pull request #2454 (2021-12-31 00:56:53)

Add telemetry for vmSettings
Adding telemetry for interesting errors in vmSettings (see _VmSettingsErrorReporter in wire.py for a description of the counters we keep track of)

Also, if the vmSettings API is not supported stop using it for 6 hours.

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
> Merging [#2454](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (5f76386) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/a7640719f663a7042f98dba5e2d6cbda57c26843?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (a764071) will **increase** coverage by `0.06%`.
> The diff coverage is `93.10%`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454/graphs/tree.svg?width=650&height=150&src=pr&token=ae90E2KYCw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)

```diff
@@             Coverage Diff             @@
##           develop    #2454      +/-   ##
===========================================
+ Coverage    71.90%   71.96%   +0.06%     
===========================================
  Files          101      101              
  Lines        14968    15015      +47     
  Branches      2370     2384      +14     
===========================================
+ Hits         10762    10805      +43     
- Misses        3727     3729       +2     
- Partials       479      481       +2     
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/utils/restutil.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi91dGlscy9yZXN0dXRpbC5weQ==) | `84.03% <33.33%> (-1.35%)` | :arrow_down: |
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `81.50% <97.50%> (+0.62%)` | :arrow_up: |
| [azurelinuxagent/common/event.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9ldmVudC5weQ==) | `85.36% <100.00%> (+0.03%)` | :arrow_up: |
| [azurelinuxagent/ga/collect\_telemetry\_events.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL2NvbGxlY3RfdGVsZW1ldHJ5X2V2ZW50cy5weQ==) | `91.43% <0.00%> (+1.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Last update [a764071...5f76386](https://codecov.io/gh/Azure/WALinuxAgent/pull/2454?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).


Patch:
diff --git a/azurelinuxagent/common/event.py b/azurelinuxagent/common/event.py
index 7149a174..b7aba5e4 100644
--- a/azurelinuxagent/common/event.py
+++ b/azurelinuxagent/common/event.py
@@ -125,6 +125,7 @@ class WALAEventOperation:
     Unknown = "Unknown"
     Update = "Update"
     VmSettings = "VmSettings"
+    VmSettingsSummary = "VmSettingsSummary"
 
 
 SHOULD_ENCODE_MESSAGE_LEN = 80
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index 890ccf02..a2b82549 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -47,7 +47,7 @@ from azurelinuxagent.common.utils.archive import StateFlusher
 from azurelinuxagent.common.utils.cryptutil import CryptUtil
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, \
-    findtext, gettext, remove_bom, get_bytes_from_pem, parse_json
+    findtext, gettext, remove_bom, get_bytes_from_pem, parse_json, format_exception
 from azurelinuxagent.common.version import AGENT_NAME, CURRENT_VERSION
 
 VERSION_INFO_URI = "http://{0}/?comp=versions"
@@ -579,9 +579,12 @@ class WireClient(object):
         self._extensions_goal_state = None  # The goal state to use for extensions; can be an ExtensionsGoalStateFromVmSettings or ExtensionsGoalStateFromExtensionsConfig
         self._vm_settings_goal_state = None  # Cached value of the most recent ExtensionsGoalStateFromVmSettings
         self._host_plugin = None
+        self._host_plugin_version = FlexibleVersion("0.0.0.0")  # Version 0 means "unknown"
+        self._host_plugin_supports_vm_settings = False
+        self._host_plugin_supports_vm_settings_next_check = datetime.now()
         self.status_blob = StatusBlob(self)
         self.goal_state_flusher = StateFlusher(conf.get_lib_dir())
-        self._vm_settings_error_reporter = _ErrorReporter(WALAEventOperation.VmSettings)
+        self._vm_settings_error_reporter = _VmSettingsErrorReporter()
 
     def get_endpoint(self):
         return self._endpoint
@@ -805,28 +808,40 @@ class WireClient(object):
 
             vm_settings_goal_state_updated = False
             if not conf.get_enable_fast_track():
-                # if Fast Track is not enabled then use extensionsConfig
+                # if Fast Track is not enabled use extensionsConfig
                 self._extensions_goal_state = self._goal_state.extensions_config
             else:
-                try:
-                    vm_settings_goal_state = self._fetch_vm_settings_goal_state(force_update=force_update)
-                    if vm_settings_goal_state is None:  # if vmSettings are None, Fast Track is not supported; use extensionsConfig
-                        self._extensions_goal_state = self._goal_state.extensions_config
-                    else:
+                if not self._host_plugin_supports_vm_settings and self._host_plugin_supports_vm_settings_next_check > datetime.now():
+                    # if vmSettings are not supported use extensionsConfig
+                    self._extensions_goal_state = self._goal_state.extensions_config
+                else:
+                    try:
+                        vm_settings_goal_state = self._fetch_vm_settings_goal_state(force_update=force_update)
+
+                        self._host_plugin_supports_vm_settings = True
+
                         if self._vm_settings_goal_state is None or self._vm_settings_goal_state.etag != vm_settings_goal_state.etag:
                             self._vm_settings_goal_state = vm_settings_goal_state
                             self._extensions_goal_state = vm_settings_goal_state
                             vm_settings_goal_state_updated = True
 
-                    if goal_state_updated or vm_settings_goal_state_updated:
-                        # compare() raises a GoalStateMismatchError if the goal states don't match
-                        ExtensionsGoalState.compare(self._goal_state.extensions_config, self._vm_settings_goal_state)
-                except Exception as error:
-                    # if there is any error on vmSettings then use extensionsConfig
-                    self._extensions_goal_state = self._goal_state.extensions_config
-                    # TODO: Once FastTrack is stable, these exceptions should be rare. Add a traceback to the error message at that point.
-                    self._vm_settings_error_reporter.report_error(ustr(error))
-                self._vm_settings_error_reporter.report_summary()
+                        if goal_state_updated or vm_settings_goal_state_updated:
+                            # compare() raises a GoalStateMismatchError if the goal states don't match
+                            ExtensionsGoalState.compare(self._goal_state.extensions_config, self._vm_settings_goal_state)
+
+                    except _VmSettingsNotSupportedError:
+                        # if vmSettings are not supported use extensionsConfig
+                        self._extensions_goal_state = self._goal_state.extensions_config
+                        # mark vmSettings as not supported for the next 6 hours
+                        self._host_plugin_supports_vm_settings = False
+                        self._host_plugin_supports_vm_settings_next_check = datetime.now() + timedelta(hours=6)
+                    except Exception as error:
+                        # if there is any errors on vmSettings then use extensionsConfig
+                        self._extensions_goal_state = self._goal_state.extensions_config
+                        # _fetch_vm_settings_goal_state() does its own detailed error reporting and raises ProtocolError; do not report those
+                        if not isinstance(error, ProtocolError):
+                            self._vm_settings_error_reporter.report_error(format_exception(error))
+                    self._vm_settings_error_reporter.report_summary()
 
             # If either goal state changed (goal_state or vm_settings_goal_state) save them
             if goal_state_updated or vm_settings_goal_state_updated:
@@ -840,26 +855,31 @@ class WireClient(object):
     def _fetch_vm_settings_goal_state(self, force_update):
         """
         Queries the vmSettings from the HostGAPlugin and returns an instance of ExtensionsGoalStateFromVmSettings.
-        Returns None if the HostGAPlugin does not support FastTrack.
+        Raises _VmSettingsNotSupportedError if the HostGAPlugin does not support FastTrack or ProtocolError on other kinds of errors.
         """
         etag = None if force_update or self._vm_settings_goal_state is None else self._vm_settings_goal_state.etag
         correlation_id = str(uuid.uuid4())
 
+        def format_message(msg):
+            return "GET vmSettings [correlation ID: {0} eTag: {1}]: {2}".format(correlation_id, etag, msg)
+
         try:
             def get_vm_settings():
                 url, headers = self.get_host_plugin().get_vm_settings_request(correlation_id)
                 if etag is not None:
                     headers['if-none-match'] = etag
-                return restutil.http_get(url, headers=headers, use_proxy=False, max_retry=1)
+                return restutil.http_get(url, headers=headers, use_proxy=False, max_retry=1, return_raw_response=True)
 
-            try:
-                response = get_vm_settings()
-            except ResourceGoneError:  # retry after refreshing the HostGAPlugin
+            self._vm_settings_error_reporter.report_request()
+
+            response = get_vm_settings()
+
+            if response.status == httpclient.GONE:  # retry after refreshing the HostGAPlugin
                 self.update_host_plugin_from_goal_state()
                 response = get_vm_settings()
 
             if response.status == httpclient.NOT_FOUND:  # the HostGAPlugin does not support FastTrack
-                return None
+                raise _VmSettingsNotSupportedError()
 
             if response.status == httpclient.NOT_MODIFIED:  # The goal state hasn't changed, return the current instance
                 return self._vm_settings_goal_state
@@ -871,28 +891,39 @@ class WireClient(object):
                 # clear any confusion produced by the poor choice of status code.
                 if response.status == httpclient.BAD_GATEWAY:
                     error_description = "[Internal error in HostGAPlugin] {0}".format(error_description)
-                raise ProtocolError("GET vmSettings [correlation ID: {0}]: {1} ({2})".format(correlation_id, response.status, error_description))
+                error_description = format_message(error_description)
+
+                if 400 <= response.status <= 499:
+                    self._vm_settings_error_reporter.report_error(error_description, _VmSettingsError.ClientError)
+                elif 500 <= response.status <= 599:
+                    self._vm_settings_error_reporter.report_error(error_description, _VmSettingsError.ServerError)
+                else:
+                    self._vm_settings_error_reporter.report_error(error_description)
+
+                raise ProtocolError(error_description)
 
             for h in response.getheaders():
                 if h[0].lower() == 'etag':
                     response_etag = h[1]
                     break
             else:  # since the vmSettings were updated, the response must include an etag
-                raise ProtocolError("The vmSettings do no include an Etag [correlation ID: {0}]".format(correlation_id))
+                message = format_message("The vmSettings response does not include an Etag header")
+                self._vm_settings_error_reporter.report_error(message)
+                raise ProtocolError(message)
 
             response_content = self.decode_config(response.read())
             vm_settings = ExtensionsGoalStateFactory.create_from_vm_settings(response_etag, response_content)
 
-            # log the HostGAPlugin version when we fetch the vmSettings for the first time
-            is_first_vm_settings = self._vm_settings_goal_state is None
-            if is_first_vm_settings:
+            # log the HostGAPlugin version
+            if vm_settings.host_ga_plugin_version != self._host_plugin_version:
+                self._host_plugin_version = vm_settings.host_ga_plugin_version
                 message = "HostGAPlugin version: {0}".format(vm_settings.host_ga_plugin_version)
                 logger.info(message)
                 add_event(op=WALAEventOperation.HostPlugin, message=message, is_success=True)
 
             # Don't support HostGAPlugin versions older than 115
             if vm_settings.host_ga_plugin_version < FlexibleVersion("1.0.8.115"):
-                return None
+                raise _VmSettingsNotSupportedError()
 
             logger.info("Fetched new vmSettings [correlation ID: {0} New eTag: {1}]", correlation_id, vm_settings.etag)
 
@@ -900,11 +931,14 @@ class WireClient(object):
 
         except ProtocolError:
             raise
-        except HttpError as http_error:
-            raise Exception("{0} [correlation ID: {1} eTag: {2}]".format(ustr(http_error), correlation_id, etag))
         except Exception as exception:
-            error = textutil.format_exception(exception)  # since this is a generic error, we format the exception to include the traceback
-            raise Exception("Error fetching vmSettings [correlation ID: {0} eTag: {1}]: {2}".format(correlation_id, etag, error))
+            if isinstance(exception, IOError) and "timed out" in ustr(exception):
+                message = format_message("Timeout")
+                self._vm_settings_error_reporter.report_error(message, _VmSettingsError.Timeout)
+            else:
+                message = format_message("Request failed: {0}".format(textutil.format_exception(exception)))
+                self._vm_settings_error_reporter.report_error(message, _VmSettingsError.RequestFailed)
+            raise ProtocolError(message)
 
     def _update_host_plugin(self, container_id, role_config_name):
         if self._host_plugin is not None:
@@ -1461,27 +1495,64 @@ class InVMArtifactsProfile(object):
         return False
 
 
-class _ErrorReporter(object):
+class _VmSettingsError(object):
+    ServerError   = 'ServerError'
+    ClientError   = 'ClientError'
+    Timeout       = 'Timeout'
+    RequestFailed = 'RequestFailed'
+
+
+class _VmSettingsErrorReporter(object):
     _MaxErrors = 5  # Max number of error reported by period
-    _Period = timedelta(hours=1)  # How often to report _errors
+    _Period = timedelta(hours=1)  # How often to report the summary
+
+    def __init__(self):
+        self._reset()
+
+    def _reset(self):
+        self._request_count = 0  # Total number of vmSettings HTTP requests
+        self._error_count = 0   # Total number of errors issuing vmSettings requests (includes all kinds of errors)
+        self._server_error_count = 0  # Count of server side errors (HTTP status in the 500s)
+        self._client_error_count = 0  # Count of client side errors (HTTP status in the 400s)
+        self._timeout_count = 0  # Count of timeouts on vmSettings requests
+        self._request_failure_count = 0  # Total count of requests that could not be issued (does not include timeouts or requests that were actually issued and failed, for example, with 500 or 400 statuses)
+        self._next_period = datetime.now() + _VmSettingsErrorReporter._Period
 
-    def __init__(self, operation):
-        self._operation = operation
-        self._error_count = 0
-        self._next_period = datetime.now() + _ErrorReporter._Period
-        self._log_message_prefix = "[{0}] [Informational only, the Agent will continue normal operation]".format(self._operation)
+    def report_request(self):
+        self._request_count += 1
 
-    def report_error(self, error):
+    def report_error(self, error, category=None):
         self._error_count += 1
-        if self._error_count <= _ErrorReporter._MaxErrors:
-            logger.info("{0} {1}", self._log_message_prefix, error)
-            add_event(op=self._operation, message=error, is_success=False, log_event=False)
+
+        if self._error_count <= _VmSettingsErrorReporter._MaxErrors:
+            logger.info("[VmSettings] [Informational only, the Agent will continue normal operation] {0}", error)
+            add_event(op=WALAEventOperation.VmSettings, message=error, is_success=False, log_event=False)
+
+        if category == _VmSettingsError.ServerError:
+            self._server_error_count += 1
+        elif category == _VmSettingsError.ClientError:
+            self._client_error_count += 1
+        elif category == _VmSettingsError.Timeout:
+            self._timeout_count += 1
+        elif category == _VmSettingsError.RequestFailed:
+            self._request_failure_count += 1
 
     def report_summary(self):
         if datetime.now() >= self._next_period:
-            self._next_period = datetime.now() + _ErrorReporter._Period
             if self._error_count > 0:
-                message = "{0} errors in the last period".format(self._error_count)
-                logger.info("{0} {1}", self._log_message_prefix, message)
-                add_event(op=self._operation, message=message, is_success=False, log_event=False)
-                self._error_count = 0
+                summary = {
+                    "requests":       self._request_count,
+                    "errors":         self._error_count,
+                    "serverErrors":   self._server_error_count,
+                    "clientErrors":   self._client_error_count,
+                    "timeouts":       self._timeout_count,
+                    "failedRequests": self._request_failure_count
+                }
+                message = json.dumps(summary)
+                logger.info("[VmSettingsSummary] {0}", message)
+                add_event(op=WALAEventOperation.VmSettingsSummary, message=message, is_success=False, log_event=False)
+            self._reset()
+
+
+class _VmSettingsNotSupportedError(ProtocolError):
+    pass
diff --git a/azurelinuxagent/common/utils/restutil.py b/azurelinuxagent/common/utils/restutil.py
index 71ffab5b..0c6d6d9a 100644
--- a/azurelinuxagent/common/utils/restutil.py
+++ b/azurelinuxagent/common/utils/restutil.py
@@ -354,7 +354,14 @@ def http_request(method,
                  max_retry=None,
                  retry_codes=None,
                  retry_delay=DELAY_IN_SECONDS,
-                 redact_data=False):
+                 redact_data=False,
+                 return_raw_response=False):
+    """
+    NOTE: This method provides some logic to handle errors in the HTTP request, including checking the HTTP status of the response
+          and handling some exceptions. If return_raw_response is set to True all the error handling will be skipped and the
+          method will return the actual HTTP response and bubble up any exceptions while issuing the request. Also note that if
+        return_raw_response is True no retries will be done.
+    """
 
     if max_retry is None:
         max_retry = DEFAULT_RETRIES
@@ -437,8 +444,12 @@ def http_request(method,
                                  proxy_host=proxy_host,
                                  proxy_port=proxy_port,
                                  redact_data=redact_data)
+
             logger.verbose("[HTTP Response] Status Code {0}", resp.status)
 
+            if return_raw_response:  # skip all error handling
+                return resp
+
             if request_failed(resp):
                 if _is_retry_status(resp.status, retry_codes=retry_codes):
                     msg = '[HTTP Retry] {0} {1} -- Status Code {2}'.format(method, url, resp.status)
@@ -465,6 +476,8 @@ def http_request(method,
             return resp
 
         except httpclient.HTTPException as e:
+            if return_raw_response:  # skip all error handling
+                raise
             clean_url = _trim_url_parameters(url)
             msg = '[HTTP Failed] {0} {1} -- HttpException {2}'.format(method, clean_url, e)
             if _is_retry_exception(e):
@@ -472,6 +485,8 @@ def http_request(method,
             break
 
         except IOError as e:
+            if return_raw_response:  # skip all error handling
+                raise
             IOErrorCounter.increment(host=host, port=port)
             clean_url = _trim_url_parameters(url)
             msg = '[HTTP Failed] {0} {1} -- IOError {2}'.format(method, clean_url, e)
@@ -485,7 +500,14 @@ def http_get(url,
              use_proxy=False,
              max_retry=None,
              retry_codes=None,
-             retry_delay=DELAY_IN_SECONDS):
+             retry_delay=DELAY_IN_SECONDS,
+             return_raw_response=False):
+    """
+    NOTE: This method provides some logic to handle errors in the HTTP request, including checking the HTTP status of the response
+          and handling some exceptions. If return_raw_response is set to True all the error handling will be skipped and the
+          method will return the actual HTTP response and bubble up any exceptions while issuing the request. Also note that if
+          return_raw_response is True no retries will be done.
+    """
 
     if max_retry is None:
         max_retry = DEFAULT_RETRIES
@@ -496,7 +518,8 @@ def http_get(url,
                         use_proxy=use_proxy,
                         max_retry=max_retry,
                         retry_codes=retry_codes,
-                        retry_delay=retry_delay)
+                        retry_delay=retry_delay,
+                        return_raw_response=return_raw_response)
 
 
 def http_head(url,


Test patch:
diff --git a/tests/protocol/test_wire.py b/tests/protocol/test_wire.py
index 326dc0bf..0ac5fc13 100644
--- a/tests/protocol/test_wire.py
+++ b/tests/protocol/test_wire.py
@@ -40,7 +40,7 @@ from azurelinuxagent.common.protocol.extensions_goal_state_from_extensions_confi
 from azurelinuxagent.common.protocol.extensions_goal_state_from_vm_settings import ExtensionsGoalStateFromVmSettings
 from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol
 from azurelinuxagent.common.protocol.wire import WireProtocol, WireClient, \
-    StatusBlob, VMStatus, EXT_CONF_FILE_NAME, _ErrorReporter
+    StatusBlob, VMStatus, EXT_CONF_FILE_NAME, _VmSettingsErrorReporter
 from azurelinuxagent.common.telemetryevent import GuestAgentExtensionEventsSchema, \
     TelemetryEventParam, TelemetryEvent
 from azurelinuxagent.common.utils import restutil, textutil
@@ -1215,10 +1215,8 @@ class UpdateGoalStateTestCase(HttpRequestPredicates, AgentTestCase):
                 self.assertEqual(e.settings[0].protectedSettings, "*** REDACTED ***", "The protected settings for {0} were not redacted".format(e.name))
 
     def test_it_should_retry_get_vm_settings_on_resource_gone_error(self):
-        """
-        Requests to the hostgaplugin incude the Container ID and the RoleConfigName as headers; when the hostgaplugin returns GONE (HTTP status 410) the agent
-        needs to get a new goal state and retry the request with updated values for the Container ID and RoleConfigName headers.
-        """
+        # Requests to the hostgaplugin incude the Container ID and the RoleConfigName as headers; when the hostgaplugin returns GONE (HTTP status 410) the agent
+        # needs to get a new goal state and retry the request with updated values for the Container ID and RoleConfigName headers.
         with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
             # Do not mock the vmSettings request at the level of azurelinuxagent.common.utils.restutil.http_request. The GONE status is handled
             # in the internal _http_request, which we mock below.
@@ -1256,26 +1254,24 @@ class UpdateGoalStateTestCase(HttpRequestPredicates, AgentTestCase):
 
         def test_error_in_http_request(test_case, mock_response, expected_error):
             def do_mock_request():
-                with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
-                    protocol.do_not_mock = lambda method, url: method == "GET" and self.is_host_plugin_vm_settings_request(url)
-
-                    def http_get_vm_settings(_method, _host, _relative_url, **_):
+                def http_get_handler(url, *_, **__):
+                    if self.is_host_plugin_vm_settings_request(url):
                         if isinstance(mock_response, Exception):
                             raise mock_response
                         return mock_response
+                    return None
 
-                    with patch("azurelinuxagent.common.utils.restutil._http_request", side_effect=http_get_vm_settings):
-                        protocol.client.update_goal_state()
+                with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS, http_get_handler=http_get_handler) as protocol:
+                    protocol.client.update_goal_state()
 
             assert_no_exception(test_case, do_mock_request, expected_error)
         #
         # We test errors different kind of errors; none of them should make update_protocol raise an exception, but all of them should be reported
         #
-        test_error_in_http_request("Internal error in the HostGAPlugin", MockHttpResponse(httpclient.BAD_GATEWAY), "Status Code 502")  # HostGAPlugin uses 502 for internal errors
+        test_error_in_http_request("Internal error in the HostGAPlugin", MockHttpResponse(httpclient.BAD_GATEWAY), "[Internal error in HostGAPlugin] [HTTP Failed] [502: None]")
         test_error_in_http_request("Arbitrary error in the request (BAD_REQUEST)", MockHttpResponse(httpclient.BAD_REQUEST), "[HTTP Failed] [400: None]")
-        test_error_in_http_request("ProtocolError during the request", ProtocolError("GENERIC PROTOCOL ERROR"), "GENERIC PROTOCOL ERROR")
         test_error_in_http_request("Generic error in the request", Exception("GENERIC REQUEST ERROR"), "GENERIC REQUEST ERROR")
-        test_error_in_http_request("Response headers with no Etag", MockHttpResponse(200, b""), "The vmSettings do no include an Etag")
+        test_error_in_http_request("Response headers with no Etag", MockHttpResponse(200, b""), "The vmSettings response does not include an Etag header")
         test_error_in_http_request("Invalid response (bad json)", MockHttpResponse(200, b"{ INVALID JSON ]", headers=[("Etag", 123)]), "Error parsing vmSettings")
 
         # Lastly, test the goal state comparison
@@ -1296,17 +1292,18 @@ class UpdateGoalStateTestCase(HttpRequestPredicates, AgentTestCase):
             protocol.set_http_handlers(http_get_handler=http_get_handler)
 
             with patch("azurelinuxagent.common.protocol.wire.add_event") as add_event:
-                for _ in range(_ErrorReporter._MaxErrors + 3):
+                for _ in range(_VmSettingsErrorReporter._MaxErrors + 3):
                     protocol.client.update_goal_state()
 
                 messages = [kwargs["message"] for _, kwargs in add_event.call_args_list if kwargs["op"] == "VmSettings"]
 
-                self.assertEqual(_ErrorReporter._MaxErrors, len(messages), "The number of errors reported is not the max allowed (got: {0})".format(messages))
+                self.assertEqual(_VmSettingsErrorReporter._MaxErrors, len(messages), "The number of errors reported is not the max allowed (got: {0})".format(messages))
 
-            with patch("azurelinuxagent.common.protocol.wire.add_event") as add_event:
-                # Reset the error reporter and verify that additional errors are reported
-                protocol.client._vm_settings_error_reporter._next_period = datetime.now()
+            # Reset the error reporter and verify that additional errors are reported
+            protocol.client._vm_settings_error_reporter._next_period = datetime.now()
+            protocol.client.update_goal_state()  # this triggers the reset
 
+            with patch("azurelinuxagent.common.protocol.wire.add_event") as add_event:
                 for _ in range(3):
                     protocol.client.update_goal_state()
 
@@ -1392,13 +1389,82 @@ class UpdateGoalStateTestCase(HttpRequestPredicates, AgentTestCase):
 
             self.assertEqual(1, compare_patcher.call_count, "ExtensionsGoalState.compare() should have been called exactly once")
 
+    def test_it_should_keep_track_of_errors_in_vm_settings_requests(self):
+        mock_response = None
+
+        def http_get_handler(url, *_, **__):
+            if self.is_host_plugin_vm_settings_request(url):
+                if isinstance(mock_response, Exception):
+                    # E0702: Raising NoneType while only classes or instances are allowed (raising-bad-type) - Disabled: we never raise None
+                    raise mock_response  # pylint: disable=raising-bad-type
+                return mock_response
+            return None
+
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS, http_get_handler=http_get_handler) as protocol:
+            mock_response = MockHttpResponse(httpclient.INTERNAL_SERVER_ERROR)
+            protocol.client.update_goal_state()
+
+            mock_response = MockHttpResponse(httpclient.BAD_REQUEST)
+            protocol.client.update_goal_state()
+            protocol.client.update_goal_state()
+
+            mock_response = IOError("timed out")
+            protocol.client.update_goal_state()
+
+            mock_response = httpclient.HTTPException()
+            protocol.client.update_goal_state()
+            protocol.client.update_goal_state()
+
+            # force the summary by resetting its period and calling update_goal_state
+            with patch("azurelinuxagent.common.protocol.wire.add_event") as add_event:
+                mock_response = None  # stop producing errors
+                protocol.client._vm_settings_error_reporter._next_period = datetime.now()
+                protocol.client.update_goal_state()
+            summary_text = [kwargs["message"] for _, kwargs in add_event.call_args_list if kwargs["op"] == "VmSettingsSummary"]
+
+            self.assertEqual(1, len(summary_text), "Exactly 1 summary should have been produced. Got: {0} ".format(summary_text))
+
+            summary = json.loads(summary_text[0])
+
+            expected = {
+                "requests":       6 + 2,  # two extra calls to update_goal_state (when creating the mock protocol and when forcing the summary)
+                "errors":         6,
+                "serverErrors":   1,
+                "clientErrors":   2,
+                "timeouts":       1,
+                "failedRequests": 2
+            }
+
+            self.assertEqual(expected, summary, "The count of errors is incorrect")
+
+    def test_it_should_stop_issuing_vm_settings_requests_when_api_is_not_supported(self):
+        with mock_wire_protocol(mockwiredata.DATA_FILE_VM_SETTINGS) as protocol:
+            def http_get_handler(url, *_, **__):
+                if self.is_host_plugin_vm_settings_request(url):
+                    return MockHttpResponse(httpclient.NOT_FOUND)  # HostGAPlugin returns 404 if the API is not supported
+                return None
+            protocol.set_http_handlers(http_get_handler=http_get_handler)
+
+            def get_vm_settings_call_count():
+                return len([url for url in protocol.get_tracked_urls() if "vmSettings" in url])
+
+            protocol.client.update_goal_state()
+            self.assertEqual(1, get_vm_settings_call_count(), "There should have been an initial call to vmSettings.")
+
+            protocol.client.update_goal_state()
+            protocol.client.update_goal_state()
+            self.assertEqual(1, get_vm_settings_call_count(), "Additional calls to update_goal_state should not have produced extra calls to vmSettings.")
+
+            # reset the vmSettings check period; this should restart the calls to the API
+            protocol.client._host_plugin_supports_vm_settings_next_check = datetime.now()
+            protocol.client.update_goal_state()
+            self.assertEqual(2, get_vm_settings_call_count(), "A second call to vmSettings was expecting after the check period has elapsed.")
+
 
 class UpdateHostPluginFromGoalStateTestCase(AgentTestCase):
     """
     Tests for WireClient.update_host_plugin_from_goal_state()
-
     """
-
     def test_it_should_update_the_host_plugin_with_or_without_incarnation_changes(self):
         with mock_wire_protocol(mockwiredata.DATA_FILE) as protocol:
             protocol.client.get_host_plugin()