Repository: Azure/WALinuxAgent
Pull request #2190 (2021-03-02 17:37:39)

Add support for required features and GoalStateAggregateStatus
<!-- DO NOT DELETE THIS TEMPLATE -->

## Description
PR for adding support for parsing Required Features list in Extension Config and also adding support for reporting Goal State Aggregate Status

Issue # <!-- if any -->
<!--
Please add an informative description that covers that changes made by the pull request. 
This checklist is used to make sure that common issues in a pull request are addressed.
This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
-->

---

### PR information
- [ ] The title of the PR is clear and informative.
- [ ] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
- [ ] New Unit tests were added for the changes made

### Quality of Code and Contribution Guidelines
- [ ] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=h1) Report
> Merging [#2190](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=desc) (dbaff6c) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/e62769e0609188a270ea953c5ea27d3402b0cd64?el=desc) (e62769e) will **increase** coverage by `0.07%`.
> The diff coverage is `93.33%`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/graphs/tree.svg?width=650&height=150&src=pr&token=ae90E2KYCw)](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #2190      +/-   ##
===========================================
+ Coverage    70.28%   70.36%   +0.07%     
===========================================
  Files           96       96              
  Lines        13433    13500      +67     
  Branches      1902     1908       +6     
===========================================
+ Hits          9442     9499      +57     
- Misses        3582     3589       +7     
- Partials       409      412       +3     
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/ga/exthandlers.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL2V4dGhhbmRsZXJzLnB5) | `86.67% <79.31%> (-0.22%)` | :arrow_down: |
| [azurelinuxagent/common/agent\_supported\_feature.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9hZ2VudF9zdXBwb3J0ZWRfZmVhdHVyZS5weQ==) | `100.00% <100.00%> (ø)` | |
| [azurelinuxagent/common/exception.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9leGNlcHRpb24ucHk=) | `98.92% <100.00%> (+0.04%)` | :arrow_up: |
| [azurelinuxagent/common/protocol/goal\_state.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9nb2FsX3N0YXRlLnB5) | `91.68% <100.00%> (+0.20%)` | :arrow_up: |
| [azurelinuxagent/common/protocol/restapi.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC9yZXN0YXBpLnB5) | `98.33% <100.00%> (+0.19%)` | :arrow_up: |
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `78.11% <100.00%> (+0.40%)` | :arrow_up: |
| [azurelinuxagent/ga/collect\_telemetry\_events.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL2NvbGxlY3RfdGVsZW1ldHJ5X2V2ZW50cy5weQ==) | `89.72% <0.00%> (-1.72%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=footer). Last update [e62769e...a707c50](https://codecov.io/gh/Azure/WALinuxAgent/pull/2190?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/azurelinuxagent/common/agent_supported_feature.py b/azurelinuxagent/common/agent_supported_feature.py
index ef5463cd..ed2044d6 100644
--- a/azurelinuxagent/common/agent_supported_feature.py
+++ b/azurelinuxagent/common/agent_supported_feature.py
@@ -98,12 +98,13 @@ def get_agent_supported_features_list_for_crp():
     List of features that the GuestAgent currently supports (like FastTrack, MultiConfig, etc).
     We need to send this list as part of Status reporting to inform CRP of all the features the agent supports.
     :return: Dict containing all CRP supported features with the key as their names and the AgentFeature object as
-             the value
+             the value if they are supported by the Agent
         Eg: {
                 MultipleExtensionsPerHandler: _MultiConfigFeature()
             }
     """
-    return __CRP_ADVERTISED_FEATURES
+
+    return dict((name, feature) for name, feature in __CRP_ADVERTISED_FEATURES.items() if feature.is_supported)
 
 
 def get_agent_supported_features_list_for_extensions():
@@ -112,9 +113,9 @@ def get_agent_supported_features_list_for_extensions():
     We need to send this list as environment variables when calling extension commands to inform Extensions of all the
     features the agent supports.
     :return: Dict containing all Extension supported features with the key as their names and the AgentFeature object as
-             the value
+             the value if the feature is supported by the Agent.
         Eg: {
                 CRPSupportedFeatureNames.ExtensionTelemetryPipeline: _ETPFeature()
             }
     """
-    return __EXTENSION_ADVERTISED_FEATURES
+    return dict((name, feature) for name, feature in __EXTENSION_ADVERTISED_FEATURES.items() if feature.is_supported)
diff --git a/azurelinuxagent/common/event.py b/azurelinuxagent/common/event.py
index bef98ac9..74f0713e 100644
--- a/azurelinuxagent/common/event.py
+++ b/azurelinuxagent/common/event.py
@@ -88,6 +88,7 @@ class WALAEventOperation:
     ExtensionTelemetryEventProcessing = "ExtensionTelemetryEventProcessing"
     FetchGoalState = "FetchGoalState"
     Firewall = "Firewall"
+    GoalStateUnsupportedFeatures = "GoalStateUnsupportedFeatures"
     HealthCheck = "HealthCheck"
     HealthObservation = "HealthObservation"
     HeartBeat = "HeartBeat"
diff --git a/azurelinuxagent/common/exception.py b/azurelinuxagent/common/exception.py
index 943214af..c4d0a685 100644
--- a/azurelinuxagent/common/exception.py
+++ b/azurelinuxagent/common/exception.py
@@ -227,6 +227,7 @@ class ServiceStoppedError(AgentError):
     """
     Error thrown when trying to access a Service which is stopped
     """
+
     def __init__(self, msg=None, inner=None):
         super(ServiceStoppedError, self).__init__(msg, inner)
 
@@ -281,3 +282,15 @@ class ExtensionErrorCodes(object):
 
     def __init__(self):
         pass
+
+
+class GoalStateAggregateStatusCodes(object):
+
+    # Success
+    Success = 0
+
+    # Unknown failure
+    GoalStateUnknownFailure = -1
+
+    # The goal state requires features that are not supported by this version of the VM agent
+    GoalStateUnsupportedRequiredFeatures = 2001
diff --git a/azurelinuxagent/common/protocol/goal_state.py b/azurelinuxagent/common/protocol/goal_state.py
index ea4b027f..6875a082 100644
--- a/azurelinuxagent/common/protocol/goal_state.py
+++ b/azurelinuxagent/common/protocol/goal_state.py
@@ -32,7 +32,7 @@ from azurelinuxagent.common.exception import ProtocolError, ExtensionConfigError
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.restapi import Cert, CertList, Extension, ExtHandler, ExtHandlerList, \
     ExtHandlerVersionUri, RemoteAccessUser, RemoteAccessUsersList, VMAgentManifest, VMAgentManifestList, \
-    VMAgentManifestUri, InVMGoalStateMetaData
+    VMAgentManifestUri, InVMGoalStateMetaData, RequiredFeature
 from azurelinuxagent.common.utils import fileutil
 from azurelinuxagent.common.utils.cryptutil import CryptUtil
 from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, findtext, getattrib, gettext
@@ -297,6 +297,7 @@ class ExtensionsConfig(object):
         self.status_upload_blob = None
         self.status_upload_blob_type = None
         self.artifacts_profile_blob = None
+        self.required_features = []
 
         if xml_text is None:
             return
@@ -319,6 +320,10 @@ class ExtensionsConfig(object):
 
         self.__parse_plugins_and_settings_and_populate_ext_handlers(xml_doc)
 
+        required_features_list = find(xml_doc, "RequiredFeatures")
+        if required_features_list is not None:
+            self._parse_required_features(required_features_list)
+
         self.status_upload_blob = findtext(xml_doc, "StatusUploadBlob")
         self.artifacts_profile_blob = findtext(xml_doc, "InVMArtifactsProfileBlob")
 
@@ -328,6 +333,12 @@ class ExtensionsConfig(object):
 
         self.in_vm_gs_metadata.parse_node(find(xml_doc, "InVMGoalStateMetaData"))
 
+    def _parse_required_features(self, required_features_list):
+        for required_feature in findall(required_features_list, "RequiredFeature"):
+            feature_name = findtext(required_feature, "Name")
+            feature_value = findtext(required_feature, "Value")
+            self.required_features.append(RequiredFeature(name=feature_name, value=feature_value))
+
     def get_redacted_xml_text(self):
         if self.xml_text is None:
             return "<None/>"
diff --git a/azurelinuxagent/common/protocol/restapi.py b/azurelinuxagent/common/protocol/restapi.py
index c74b70f8..619620d5 100644
--- a/azurelinuxagent/common/protocol/restapi.py
+++ b/azurelinuxagent/common/protocol/restapi.py
@@ -18,12 +18,13 @@
 #
 
 import socket
+import time
 from datetime import datetime, timedelta
 
+from azurelinuxagent.common.datacontract import DataContract, DataContractList
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.utils.textutil import getattrib
 from azurelinuxagent.common.version import DISTRO_VERSION, DISTRO_NAME, CURRENT_VERSION
-from azurelinuxagent.common.datacontract import DataContract, DataContractList
 
 
 class VMInfo(DataContract):
@@ -81,6 +82,13 @@ class VMAgentManifestList(DataContract):
         self.vmAgentManifests = DataContractList(VMAgentManifest)
 
 
+class RequiredFeature(DataContract):
+    def __init__(self, name, value=None):
+        self.name = name
+        # As per the docs, this is a reserved field and not currently in use.
+        self.value = value
+
+
 class Extension(DataContract):
     """
     The runtime settings associated with a Handler
@@ -90,7 +98,6 @@ class Extension(DataContract):
         ExtensionConfig.xml
         Eg: <extensionName>.1.settings, <extensionName>.2.settings
     """
-
     def __init__(self,
                  name=None,
                  sequenceNumber=None,
@@ -267,7 +274,7 @@ class ExtHandlerStatus(DataContract):
 
 
 class VMAgentStatus(DataContract):
-    def __init__(self, status=None, message=None):
+    def __init__(self, status=None, message=None, gs_aggregate_status=None):
         self.status = status
         self.message = message
         self.hostname = socket.gethostname()
@@ -275,11 +282,30 @@ class VMAgentStatus(DataContract):
         self.osname = DISTRO_NAME
         self.osversion = DISTRO_VERSION
         self.extensionHandlers = DataContractList(ExtHandlerStatus)
+        self.vm_artifacts_aggregate_status = VMArtifactsAggregateStatus(gs_aggregate_status)
 
 
 class VMStatus(DataContract):
-    def __init__(self, status, message):
-        self.vmAgent = VMAgentStatus(status=status, message=message)
+    def __init__(self, status, message, gs_aggregate_status=None):
+        self.vmAgent = VMAgentStatus(status=status, message=message, gs_aggregate_status=gs_aggregate_status)
+
+
+class GoalStateAggregateStatus(DataContract):
+    def __init__(self, status=None, seq_no=-1, message="", code=None):
+        self.message = message
+        self.in_svd_seq_no = seq_no
+        self.status = status
+        self.code = code
+        self.__utc_timestamp = time.gmtime()
+
+    @property
+    def processed_time(self):
+        return self.__utc_timestamp
+
+
+class VMArtifactsAggregateStatus(DataContract):
+    def __init__(self, gs_aggregate_status=None):
+        self.goal_state_aggregate_status = gs_aggregate_status
 
 
 class RemoteAccessUser(DataContract):
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index 9b890ae1..e85dc9fa 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -126,6 +126,9 @@ class WireProtocol(DataContract):
     def get_in_vm_gs_metadata(self):
         return self.client.get_ext_conf().in_vm_gs_metadata
 
+    def get_required_features(self):
+        return self.client.get_ext_conf().required_features
+
     def get_vmagent_manifests(self):
         goal_state = self.client.get_goal_state()
         ext_conf = self.client.get_ext_conf()
@@ -285,15 +288,23 @@ def ga_status_to_guest_info(ga_status):
     return v1_ga_guest_info
 
 
-def ga_status_to_v1(ga_status):
-    formatted_msg = {
-        'lang': 'en-US',
-        'message': ga_status.message
+def __get_formatted_msg_for_status_reporting(msg, lang="en-US"):
+    return {
+        'lang': lang,
+        'message': msg
     }
+
+
+def _get_utc_timestamp_for_status_reporting(time_format="%Y-%m-%dT%H:%M:%SZ", timestamp=None):
+    timestamp = time.gmtime() if timestamp is None else timestamp
+    return time.strftime(time_format, timestamp)
+
+
+def ga_status_to_v1(ga_status):
     v1_ga_status = {
         "version": ga_status.version,
         "status": ga_status.status,
-        "formattedMessage": formatted_msg
+        "formattedMessage": __get_formatted_msg_for_status_reporting(ga_status.message)
     }
     return v1_ga_status
 
@@ -305,10 +316,7 @@ def ext_substatus_to_v1(sub_status_list):
             "name": substatus.name,
             "status": substatus.status,
             "code": substatus.code,
-            "formattedMessage": {
-                "lang": "en-US",
-                "message": substatus.message
-            }
+            "formattedMessage": __get_formatted_msg_for_status_reporting(substatus.message)
         }
         status_list.append(status)
     return status_list
@@ -317,7 +325,7 @@ def ext_substatus_to_v1(sub_status_list):
 def ext_status_to_v1(ext_name, ext_status):
     if ext_status is None:
         return None
-    timestamp = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+    timestamp = _get_utc_timestamp_for_status_reporting()
     v1_sub_status = ext_substatus_to_v1(ext_status.substatusList)
     v1_ext_status = {
         "status": {
@@ -326,10 +334,7 @@ def ext_status_to_v1(ext_name, ext_status):
             "operation": ext_status.operation,
             "status": ext_status.status,
             "code": ext_status.code,
-            "formattedMessage": {
-                "lang": "en-US",
-                "message": ext_status.message
-            }
+            "formattedMessage": __get_formatted_msg_for_status_reporting(ext_status.message)
         },
         "version": 1.0,
         "timestampUTC": timestamp
@@ -339,7 +344,7 @@ def ext_status_to_v1(ext_name, ext_status):
     return v1_ext_status
 
 
-def ext_handler_status_to_v1(handler_status, ext_statuses, timestamp):  # pylint: disable=W0613
+def ext_handler_status_to_v1(handler_status, ext_statuses):
     v1_handler_status = {
         'handlerVersion': handler_status.version,
         'handlerName': handler_status.name,
@@ -348,10 +353,7 @@ def ext_handler_status_to_v1(handler_status, ext_statuses, timestamp):  # pylint
         'useExactVersion': True
     }
     if handler_status.message is not None:
-        v1_handler_status["formattedMessage"] = {
-            "lang": "en-US",
-            "message": handler_status.message
-        }
+        v1_handler_status["formattedMessage"] = __get_formatted_msg_for_status_reporting(handler_status.message)
 
     if len(handler_status.extensions) > 0:
         # Currently, no more than one extension per handler
@@ -366,15 +368,36 @@ def ext_handler_status_to_v1(handler_status, ext_statuses, timestamp):  # pylint
     return v1_handler_status
 
 
+def vm_artifacts_aggregate_status_to_v1(vm_artifacts_aggregate_status):
+    gs_aggregate_status = vm_artifacts_aggregate_status.goal_state_aggregate_status
+    if gs_aggregate_status is None:
+        return None
+
+    v1_goal_state_aggregate_status = {
+        "formattedMessage": __get_formatted_msg_for_status_reporting(gs_aggregate_status.message),
+        "timestampUTC": _get_utc_timestamp_for_status_reporting(timestamp=gs_aggregate_status.processed_time),
+        "inSvdSeqNo": gs_aggregate_status.in_svd_seq_no,
+        "status": gs_aggregate_status.status,
+        "code": gs_aggregate_status.code
+    }
+
+    v1_artifact_aggregate_status = {
+        "goalStateAggregateStatus": v1_goal_state_aggregate_status
+    }
+    return v1_artifact_aggregate_status
+
+
 def vm_status_to_v1(vm_status, ext_statuses):
-    timestamp = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+    timestamp = _get_utc_timestamp_for_status_reporting()
 
     v1_ga_guest_info = ga_status_to_guest_info(vm_status.vmAgent)
     v1_ga_status = ga_status_to_v1(vm_status.vmAgent)
+    v1_vm_artifact_aggregate_status = vm_artifacts_aggregate_status_to_v1(
+        vm_status.vmAgent.vm_artifacts_aggregate_status)
     v1_handler_status_list = []
     for handler_status in vm_status.vmAgent.extensionHandlers:
         v1_handler_status = ext_handler_status_to_v1(handler_status,
-                                                     ext_statuses, timestamp)
+                                                     ext_statuses)
         if v1_handler_status is not None:
             v1_handler_status_list.append(v1_handler_status)
 
@@ -382,6 +405,10 @@ def vm_status_to_v1(vm_status, ext_statuses):
         'guestAgentStatus': v1_ga_status,
         'handlerAggregateStatus': v1_handler_status_list
     }
+
+    if v1_vm_artifact_aggregate_status is not None:
+        v1_agg_status['vmArtifactsAggregateStatus'] = v1_vm_artifact_aggregate_status
+
     v1_vm_status = {
         'version': '1.1',
         'timestampUTC': timestamp,
@@ -391,13 +418,12 @@ def vm_status_to_v1(vm_status, ext_statuses):
 
     supported_features = []
     for _, feature in get_agent_supported_features_list_for_crp().items():
-        if feature.is_supported:
-            supported_features.append(
-                {
-                    "Key": feature.name,
-                    "Value": feature.version
-                }
-            )
+        supported_features.append(
+            {
+                "Key": feature.name,
+                "Value": feature.version
+            }
+        )
     if supported_features:
         v1_vm_status["supportedFeatures"] = supported_features
 
@@ -451,7 +477,7 @@ class StatusBlob(object):
         return {
             "Content-Length": ustr(blob_size),
             "x-ms-blob-type": "BlockBlob",
-            "x-ms-date": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "x-ms-date": _get_utc_timestamp_for_status_reporting(),
             "x-ms-version": self.__class__.__storage_version__
         }
 
@@ -468,14 +494,14 @@ class StatusBlob(object):
             "Content-Length": "0",
             "x-ms-blob-content-length": ustr(blob_size),
             "x-ms-blob-type": "PageBlob",
-            "x-ms-date": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "x-ms-date": _get_utc_timestamp_for_status_reporting(),
             "x-ms-version": self.__class__.__storage_version__
         }
 
     def get_page_blob_page_headers(self, start, end):
         return {
             "Content-Length": ustr(end - start),
-            "x-ms-date": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "x-ms-date": _get_utc_timestamp_for_status_reporting(),
             "x-ms-range": "bytes={0}-{1}".format(start, end - 1),
             "x-ms-page-write": "update",
             "x-ms-version": self.__class__.__storage_version__
diff --git a/azurelinuxagent/ga/exthandlers.py b/azurelinuxagent/ga/exthandlers.py
index ca39ddf0..6b79c3ed 100644
--- a/azurelinuxagent/ga/exthandlers.py
+++ b/azurelinuxagent/ga/exthandlers.py
@@ -37,17 +37,17 @@ import azurelinuxagent.common.logger as logger
 import azurelinuxagent.common.utils.fileutil as fileutil
 import azurelinuxagent.common.version as version
 from azurelinuxagent.common.agent_supported_feature import get_agent_supported_features_list_for_extensions, \
-    SupportedFeatureNames, get_supported_feature_by_name
+    SupportedFeatureNames, get_supported_feature_by_name, get_agent_supported_features_list_for_crp
 from azurelinuxagent.common.cgroupconfigurator import CGroupConfigurator
 from azurelinuxagent.common.datacontract import get_properties, set_properties
 from azurelinuxagent.common.errorstate import ErrorState
 from azurelinuxagent.common.event import add_event, elapsed_milliseconds, report_event, WALAEventOperation, \
     add_periodic, EVENTS_DIRECTORY
 from azurelinuxagent.common.exception import ExtensionDownloadError, ExtensionError, ExtensionErrorCodes, \
-    ExtensionOperationError, ExtensionUpdateError, ProtocolError, ProtocolNotFoundError, ExtensionConfigError
+    ExtensionOperationError, ExtensionUpdateError, ProtocolError, ProtocolNotFoundError, ExtensionConfigError, GoalStateAggregateStatusCodes
 from azurelinuxagent.common.future import ustr, is_file_not_found_error
 from azurelinuxagent.common.protocol.restapi import ExtensionStatus, ExtensionSubStatus, ExtHandler, ExtHandlerStatus, \
-    VMStatus
+    VMStatus, GoalStateAggregateStatus
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 from azurelinuxagent.common.version import AGENT_NAME, CURRENT_VERSION, DISTRO_NAME, DISTRO_VERSION, \
     GOAL_STATE_AGENT_VERSION, PY_VERSION_MAJOR, PY_VERSION_MICRO, PY_VERSION_MINOR
@@ -243,6 +243,23 @@ class ExtensionRequestedState(object):
     Uninstall = u"uninstall"
 
 
+class GoalStateStatus(object):
+    """
+    This is an Enum to define the State of the GoalState as a whole. This is reported as part of the
+    'vmArtifactsAggregateStatus.goalStateAggregateStatus' in the status blob.
+    Note: not to be confused with the State of the ExtHandler which reported as part of 'handlerAggregateStatus'
+    """
+    Success = "Success"
+    Failed = "Failed"
+
+    # This is used when we initialize a new Goal State, before we start processing it.
+    # It doesn't hold much value for us until we move status reporting to a separate thread.
+    Initialize = "Initialize"
+
+    # The following field is not used now but would be needed once Status reporting is moved to a separate thread.
+    Transitioning = "Transitioning"
+
+
 def get_exthandlers_handler(protocol):
     return ExtHandlersHandler(protocol)
 
@@ -265,6 +282,13 @@ class ExtHandlersHandler(object):
         self.last_etag = None
         self.log_report = False
         self.log_process = False
+        # The GoalState Aggregate status needs to report the last status of the GoalState. Since we only process
+        # extensions on incarnation change, we need to maintain its state.
+        # Setting the status as Initialize here. This would be overridden as soon as the first GoalState is processed
+        # (once self._extension_processing_allowed() is True).
+        self.__gs_aggregate_status = GoalStateAggregateStatus(status=GoalStateStatus.Initialize, seq_no="-1",
+                                                              code=GoalStateAggregateStatusCodes.Success,
+                                                              message="Initializing new GoalState")
 
         self.report_status_error_state = ErrorState()
 
@@ -303,7 +327,8 @@ class ExtHandlersHandler(object):
                 logger.info(
                     "ProcessGoalState started [Incarnation: {0}; Activity Id: {1}; Correlation Id: {2}; GS Creation Time: {3}]".format(
                         etag, activity_id, correlation_id, gs_creation_time))
-                self.handle_ext_handlers(etag)
+
+                self.__process_and_handle_extensions(etag)
                 self.last_etag = etag
 
             self.report_ext_handlers_status()
@@ -319,6 +344,46 @@ class ExtHandlersHandler(object):
                       message=detailed_msg)
             return
 
+    def __get_unsupported_features(self):
+        required_features = self.protocol.get_required_features()
+        supported_features = get_agent_supported_features_list_for_crp()
+        return [feature.name for feature in required_features if feature.name not in supported_features]
+
+    def __process_and_handle_extensions(self, etag):
+        try:
+            # Verify we satisfy all required features, if any. If not, report failure here itself, no need to process anything further.
+            unsupported_features = self.__get_unsupported_features()
+            if any(unsupported_features):
+                msg = "Failing GS incarnation: {0} as Unsupported features found: {1}".format(etag, ', '.join(
+                    unsupported_features))
+                logger.warn(msg)
+                self.__gs_aggregate_status = GoalStateAggregateStatus(status=GoalStateStatus.Failed, seq_no=etag,
+                                                                      code=GoalStateAggregateStatusCodes.GoalStateUnsupportedRequiredFeatures,
+                                                                      message=msg)
+                add_event(AGENT_NAME,
+                          version=CURRENT_VERSION,
+                          op=WALAEventOperation.GoalStateUnsupportedFeatures,
+                          is_success=False,
+                          message=msg,
+                          log_event=False)
+            else:
+                self.handle_ext_handlers(etag)
+                self.__gs_aggregate_status = GoalStateAggregateStatus(status=GoalStateStatus.Success, seq_no=etag,
+                                                                      code=GoalStateAggregateStatusCodes.Success,
+                                                                      message="GoalState executed successfully")
+        except Exception as error:
+            msg = "Unexpected error when processing goal state: {0}; {1}".format(ustr(error), traceback.format_exc())
+            self.__gs_aggregate_status = GoalStateAggregateStatus(status=GoalStateStatus.Failed, seq_no=etag,
+                                                                  code=GoalStateAggregateStatusCodes.GoalStateUnknownFailure,
+                                                                  message=msg)
+            logger.warn(msg)
+            add_event(AGENT_NAME,
+                      version=CURRENT_VERSION,
+                      op=WALAEventOperation.ExtensionProcessing,
+                      is_success=False,
+                      message=msg,
+                      log_event=False)
+
     @staticmethod
     def get_ext_handler_instance_from_path(name, path, protocol, skip_handlers=None):
         if not os.path.isdir(path) or re.match(HANDLER_NAME_PATTERN, name) is None:
@@ -661,18 +726,19 @@ class ExtHandlersHandler(object):
         """
         Go through handler_state dir, collect and report status
         """
-        vm_status = VMStatus(status="Ready", message="Guest Agent is running")
+        vm_status = VMStatus(status="Ready", message="Guest Agent is running",
+                             gs_aggregate_status=self.__gs_aggregate_status)
         if self.ext_handlers is not None:
             for ext_handler in self.ext_handlers.extHandlers:
                 try:
                     self.report_ext_handler_status(vm_status, ext_handler)
-                except ExtensionError as e:
+                except ExtensionError as error:
                     add_event(
                         AGENT_NAME,
                         version=CURRENT_VERSION,
                         op=WALAEventOperation.ExtensionProcessing,
                         is_success=False,
-                        message=ustr(e))
+                        message=ustr(error))
 
         logger.verbose("Report vm agent status")
         try:
@@ -680,13 +746,13 @@ class ExtHandlersHandler(object):
             if self.log_report:
                 logger.verbose("Completed vm agent status report")
             self.report_status_error_state.reset()
-        except ProtocolNotFoundError as e:
+        except ProtocolNotFoundError as error:
             self.report_status_error_state.incr()
-            message = "Failed to report vm agent status: {0}".format(e)
+            message = "Failed to report vm agent status: {0}".format(error)
             logger.verbose(message)
-        except ProtocolError as e:
+        except ProtocolError as error:
             self.report_status_error_state.incr()
-            message = "Failed to report vm agent status: {0}".format(e)
+            message = "Failed to report vm agent status: {0}".format(error)
             add_event(AGENT_NAME,
                       version=CURRENT_VERSION,
                       op=WALAEventOperation.ExtensionProcessing,
@@ -770,7 +836,7 @@ class ExtHandlerInstance(object):
         try:
             if os.stat(filename).st_size <= max_size:
                 return
-            
+
             with open(filename, "rb") as existing_file:
                 existing_file.seek(-1 * max_size, 2)
                 _ = existing_file.readline()
@@ -786,7 +852,7 @@ class ExtHandlerInstance(object):
                 # this just means that no extension with self.ext_handler.name is
                 # installed.
                 return
-            
+
             logger.error("Exception occurred while attempting to truncate CommandExecution.log "
                 "for extension {0}. Exception is: {1}", self.ext_handler.name, e)
 
@@ -1409,13 +1475,12 @@ class ExtHandlerInstance(object):
 
                 supported_features = []
                 for _, feature in get_agent_supported_features_list_for_extensions().items():
-                    if feature.is_supported:
-                        supported_features.append(
-                            {
-                                "Key": feature.name,
-                                "Value": feature.version
-                            }
-                        )
+                    supported_features.append(
+                        {
+                            "Key": feature.name,
+                            "Value": feature.version
+                        }
+                    )
                 if supported_features:
                     env.update({
                         ExtCommandEnvVariable.ExtensionSupportedFeatures: json.dumps(supported_features)


Test patch:
diff --git a/tests/common/test_agent_supported_feature.py b/tests/common/test_agent_supported_feature.py
index fcae9ec6..cf367f90 100644
--- a/tests/common/test_agent_supported_feature.py
+++ b/tests/common/test_agent_supported_feature.py
@@ -14,17 +14,24 @@
 #
 # Requires Python 2.6+ and Openssl 1.0+
 #
+
 from azurelinuxagent.common.agent_supported_feature import SupportedFeatureNames, \
     get_agent_supported_features_list_for_crp, get_supported_feature_by_name, \
     get_agent_supported_features_list_for_extensions
-from tests.tools import AgentTestCase
+from tests.tools import AgentTestCase, patch
 
 
 class TestAgentSupportedFeature(AgentTestCase):
 
     def test_it_should_return_features_properly(self):
-        self.assertIn(SupportedFeatureNames.MultiConfig, get_agent_supported_features_list_for_crp(),
-                      "Multi-config should be fetched in crp_supported_features")
+        with patch("azurelinuxagent.common.agent_supported_feature._MultiConfigFeature.is_supported", True):
+            self.assertIn(SupportedFeatureNames.MultiConfig, get_agent_supported_features_list_for_crp(),
+                          "Multi-config should be fetched in crp_supported_features")
+
+        with patch("azurelinuxagent.common.agent_supported_feature._MultiConfigFeature.is_supported", False):
+            self.assertNotIn(SupportedFeatureNames.MultiConfig, get_agent_supported_features_list_for_crp(),
+                             "Multi-config should not be fetched in crp_supported_features as not supported")
+
         self.assertEqual(SupportedFeatureNames.MultiConfig,
                          get_supported_feature_by_name(SupportedFeatureNames.MultiConfig).name,
                          "Invalid/Wrong feature returned")
@@ -34,8 +41,14 @@ class TestAgentSupportedFeature(AgentTestCase):
             get_supported_feature_by_name("ABC")
 
     def test_it_should_return_extension_supported_features_properly(self):
-        self.assertIn(SupportedFeatureNames.ExtensionTelemetryPipeline,
-                      get_agent_supported_features_list_for_extensions(), "ETP should be in supported features list")
+        with patch("azurelinuxagent.common.agent_supported_feature._ETPFeature.is_supported", True):
+            self.assertIn(SupportedFeatureNames.ExtensionTelemetryPipeline,
+                          get_agent_supported_features_list_for_extensions(), "ETP should be in supported features list")
+
+        with patch("azurelinuxagent.common.agent_supported_feature._ETPFeature.is_supported", False):
+            self.assertNotIn(SupportedFeatureNames.ExtensionTelemetryPipeline,
+                             get_agent_supported_features_list_for_extensions(),
+                             "ETP should not be in supported features list")
 
         self.assertEqual(SupportedFeatureNames.ExtensionTelemetryPipeline,
                          get_supported_feature_by_name(SupportedFeatureNames.ExtensionTelemetryPipeline).name,
diff --git a/tests/data/wire/ext_conf_required_features.xml b/tests/data/wire/ext_conf_required_features.xml
new file mode 100644
index 00000000..6b9b23c1
--- /dev/null
+++ b/tests/data/wire/ext_conf_required_features.xml
@@ -0,0 +1,40 @@
+<Extensions version="1.0.0.0" goalStateIncarnation="9">
+	<GuestAgentExtension xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
+		<GAFamilies>
+			<GAFamily>
+				<Name>Prod</Name>
+				<Uris>
+					<Uri>http://mock-goal-state/manifest_of_ga.xml</Uri>
+				</Uris>
+			</GAFamily>
+			<GAFamily>
+				<Name>Test</Name>
+				<Uris>
+					<Uri>http://mock-goal-state/manifest_of_ga.xml</Uri>
+				</Uris>
+			</GAFamily>
+		</GAFamilies>
+	</GuestAgentExtension>
+	<RequiredFeatures>
+		<RequiredFeature>
+			<Name>TestRequiredFeature1</Name>
+		</RequiredFeature>
+		<RequiredFeature>
+			<Name>TestRequiredFeature2</Name>
+		</RequiredFeature>
+		<RequiredFeature>
+			<Name>TestRequiredFeature3</Name>
+			<Value>3.0</Value>
+		</RequiredFeature>
+	</RequiredFeatures>
+	<Plugins>
+		<Plugin name="OSTCExtensions.ExampleHandlerLinux" version="1.0.0" location="http://mock-goal-state/rdfepirv2hknprdstr03.blob.core.windows.net/b01058962be54ceca550a390fa5ff064/Microsoft.OSTCExtensions_ExampleHandlerLinux_asiaeast_manifest.xml" config="" state="enabled" autoUpgrade="false" failoverlocation="http://mock-goal-state/rdfepirv2hknprdstr04.blob.core.windows.net/b01058962be54ceca550a390fa5ff064/Microsoft.OSTCExtensions_ExampleHandlerLinux_asiaeast_manifest.xml" runAsStartupTask="false" isJson="true" />
+	</Plugins>
+	<PluginSettings>
+		<Plugin name="OSTCExtensions.ExampleHandlerLinux" version="1.0.0">
+			<RuntimeSettings seqNo="0">{"runtimeSettings":[{"handlerSettings":{"protectedSettingsCertThumbprint":"4037FBF5F1F3014F99B5D6C7799E9B20E6871CB3","protectedSettings":"MIICWgYJK","publicSettings":{"foo":"bar"}}}]}</RuntimeSettings>
+		</Plugin>
+	</PluginSettings>
+	<StatusUploadBlob statusBlobType="BlockBlob">https://test.blob.core.windows.net/vhds/test-cs12.test-cs12.test-cs12.status?sr=b&amp;sp=rw&amp;se=9999-01-01&amp;sk=key1&amp;sv=2014-02-14&amp;sig=hfRh7gzUE7sUtYwke78IOlZOrTRCYvkec4hGZ9zZzXo</StatusUploadBlob>
+</Extensions>
+
diff --git a/tests/ga/test_extension.py b/tests/ga/test_extension.py
index 1b1966ba..3b8b3bc3 100644
--- a/tests/ga/test_extension.py
+++ b/tests/ga/test_extension.py
@@ -38,7 +38,7 @@ from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 from azurelinuxagent.common.version import PY_VERSION_MAJOR, PY_VERSION_MINOR, PY_VERSION_MICRO, AGENT_NAME, \
     GOAL_STATE_AGENT_VERSION, CURRENT_VERSION, DISTRO_NAME, DISTRO_VERSION
 from azurelinuxagent.common.exception import ResourceGoneError, ExtensionDownloadError, ProtocolError, \
-    ExtensionErrorCodes, ExtensionError
+    ExtensionErrorCodes, ExtensionError, GoalStateAggregateStatusCodes
 from azurelinuxagent.common.protocol.restapi import Extension, ExtHandler, ExtHandlerStatus, \
     ExtensionStatus
 from azurelinuxagent.common.protocol.wire import WireProtocol, InVMArtifactsProfile
@@ -46,7 +46,7 @@ from azurelinuxagent.common.utils.restutil import KNOWN_WIRESERVER_IP
 
 from azurelinuxagent.ga.exthandlers import ExtHandlersHandler, ExtHandlerInstance, migrate_handler_state, \
     get_exthandlers_handler, AGENT_STATUS_FILE, ExtCommandEnvVariable, HandlerManifest, NOT_RUN, \
-    ValidHandlerStatus, HANDLER_COMPLETE_NAME_PATTERN, HandlerEnvironment, ExtensionRequestedState
+    ValidHandlerStatus, HANDLER_COMPLETE_NAME_PATTERN, HandlerEnvironment, ExtensionRequestedState, GoalStateStatus
 
 from tests.protocol import mockwiredata
 from tests.protocol.mocks import mock_wire_protocol, HttpRequestPredicates
@@ -2309,6 +2309,66 @@ class TestExtension(AgentTestCase):
                 self.assertIn("%s=%s" % (ExtCommandEnvVariable.UninstallReturnCode, exit_code), install_kwargs['message'])
                 self.assertIn("%s=%s" % (ExtCommandEnvVariable.UninstallReturnCode, exit_code), enable_kwargs['message'])
 
+    def test_it_should_persist_goal_state_aggregate_status_until_new_incarnation(self, mock_get, mock_crypt_util, *args):
+        test_data = mockwiredata.WireProtocolData(mockwiredata.DATA_FILE)
+        exthandlers_handler, protocol = self._create_mock(test_data, mock_get, mock_crypt_util, *args)
+
+        exthandlers_handler.run()
+        self._assert_handler_status(protocol.report_vm_status, "Ready", expected_ext_count=1, version="1.0.0")
+        args, _ = protocol.report_vm_status.call_args
+        gs_aggregate_status = args[0].vmAgent.vm_artifacts_aggregate_status.goal_state_aggregate_status
+        self.assertIsNotNone(gs_aggregate_status, "Goal State Aggregate status not reported")
+        self.assertEqual(gs_aggregate_status.status, GoalStateStatus.Success, "Wrong status reported")
+        self.assertEqual(gs_aggregate_status.in_svd_seq_no, "1", "Incorrect seq no")
+
+        # Running handler again without incarnation change should report the same gs_aggregate_status
+        for _ in range(5):
+            exthandlers_handler.run()
+            args, _ = protocol.report_vm_status.call_args
+            self.assertEqual(gs_aggregate_status,
+                             args[0].vmAgent.vm_artifacts_aggregate_status.goal_state_aggregate_status,
+                             "Aggregate status different")
+
+        # Update incarnation and ensure the gs_aggregate_status is modified too
+        test_data.set_incarnation(2)
+        protocol.client.update_goal_state()
+        exthandlers_handler.run()
+        self._assert_handler_status(protocol.report_vm_status, "Ready", expected_ext_count=1, version="1.0.0")
+        args, _ = protocol.report_vm_status.call_args
+        new_gs_aggregate_status = args[0].vmAgent.vm_artifacts_aggregate_status.goal_state_aggregate_status
+        self.assertIsNotNone(new_gs_aggregate_status, "New Goal State Aggregate status not reported")
+        self.assertNotEqual(gs_aggregate_status, new_gs_aggregate_status, "The gs_aggregate_status should be different")
+        self.assertEqual(new_gs_aggregate_status.status, GoalStateStatus.Success, "Wrong status reported")
+        self.assertEqual(new_gs_aggregate_status.in_svd_seq_no, "2", "Incorrect seq no")
+
+    def test_it_should_parse_required_features_properly(self, mock_get, mock_crypt_util, *args):
+        test_data = mockwiredata.WireProtocolData(mockwiredata.DATA_FILE_REQUIRED_FEATURES)
+        _, protocol = self._create_mock(test_data, mock_get, mock_crypt_util, *args)
+
+        required_features = protocol.get_required_features()
+        self.assertEqual(3, len(required_features), "Incorrect features parsed")
+        for i, feature in enumerate(required_features):
+            self.assertEqual(feature.name, "TestRequiredFeature{0}".format(i+1), "Name mismatch")
+            if i < 2:
+                self.assertIsNone(feature.value, "Value not present")
+            else:
+                self.assertEqual(feature.value, "3.0", "Value mismatch")
+
+    def test_it_should_fail_goal_state_if_required_features_not_supported(self, mock_get, mock_crypt_util, *args):
+        test_data = mockwiredata.WireProtocolData(mockwiredata.DATA_FILE_REQUIRED_FEATURES)
+        exthandlers_handler, protocol = self._create_mock(test_data, mock_get, mock_crypt_util, *args)
+
+        exthandlers_handler.run()
+        args, _ = protocol.report_vm_status.call_args
+        gs_aggregate_status = args[0].vmAgent.vm_artifacts_aggregate_status.goal_state_aggregate_status
+        self.assertEqual(0, len(args[0].vmAgent.extensionHandlers), "No extensions should be reported")
+        self.assertIsNotNone(gs_aggregate_status, "GS Aggregagte status should be reported")
+        self.assertEqual(gs_aggregate_status.status, GoalStateStatus.Failed, "GS should be failed")
+        self.assertEqual(gs_aggregate_status.code, GoalStateAggregateStatusCodes.GoalStateUnsupportedRequiredFeatures,
+                         "Incorrect error code set properly for GS failure")
+        self.assertEqual(gs_aggregate_status.in_svd_seq_no, "1", "Sequence Number is wrong")
+
+
 @patch("azurelinuxagent.common.protocol.wire.CryptUtil")
 @patch("azurelinuxagent.common.utils.restutil.http_get")
 class TestExtensionSequencing(AgentTestCase):
diff --git a/tests/ga/test_exthandlers.py b/tests/ga/test_exthandlers.py
index 0b2b1b72..4893dca3 100644
--- a/tests/ga/test_exthandlers.py
+++ b/tests/ga/test_exthandlers.py
@@ -762,8 +762,8 @@ print("Found Feature %s: %s" % ("{1}", found))
 
         # It should include all supported features and pass it as Environment Variable to extensions
         test_supported_features = {test_name: TestFeature(name=test_name, version=test_version, supported=True)}
-        with patch("azurelinuxagent.ga.exthandlers.get_agent_supported_features_list_for_extensions",
-                   return_value=test_supported_features):
+        with patch("azurelinuxagent.common.agent_supported_feature.__EXTENSION_ADVERTISED_FEATURES",
+                   test_supported_features):
             output = self.ext_handler_instance.launch_command(command)
 
             self.assertIn("[stdout]\nFound Feature {0}: True".format(test_name), output, "Feature not found")
@@ -773,16 +773,16 @@ print("Found Feature %s: %s" % ("{1}", found))
             test_name: TestFeature(name=test_name, version=test_version, supported=False),
             "testFeature": TestFeature(name="testFeature", version="1.2.1", supported=True)
         }
-        with patch("azurelinuxagent.ga.exthandlers.get_agent_supported_features_list_for_extensions",
-                   return_value=test_supported_features):
+        with patch("azurelinuxagent.common.agent_supported_feature.__EXTENSION_ADVERTISED_FEATURES",
+                   test_supported_features):
             output = self.ext_handler_instance.launch_command(command)
 
             self.assertIn("[stdout]\nFound Feature {0}: False".format(test_name), output, "Feature wrongfully found")
 
         # It should not include the SupportedFeatures Key in Environment variables if no features supported
         test_supported_features = {test_name: TestFeature(name=test_name, version=test_version, supported=False)}
-        with patch("azurelinuxagent.ga.exthandlers.get_agent_supported_features_list_for_extensions",
-                   return_value=test_supported_features):
+        with patch("azurelinuxagent.common.agent_supported_feature.__EXTENSION_ADVERTISED_FEATURES",
+                   test_supported_features):
             output = self.ext_handler_instance.launch_command(command)
 
             self.assertIn(
diff --git a/tests/protocol/mockwiredata.py b/tests/protocol/mockwiredata.py
index 006634f8..bf5a2188 100644
--- a/tests/protocol/mockwiredata.py
+++ b/tests/protocol/mockwiredata.py
@@ -100,6 +100,9 @@ DATA_FILE_REMOTE_ACCESS["remote_access"] = "wire/remote_access_single_account.xm
 DATA_FILE_PLUGIN_SETTINGS_MISMATCH = DATA_FILE.copy()
 DATA_FILE_PLUGIN_SETTINGS_MISMATCH["ext_conf"] = "wire/invalid_config/ext_conf_plugin_settings_version_mismatch.xml"
 
+DATA_FILE_REQUIRED_FEATURES = DATA_FILE.copy()
+DATA_FILE_REQUIRED_FEATURES["ext_conf"] = "wire/ext_conf_required_features.xml"
+
 
 class WireProtocolData(object):
     def __init__(self, data_files=None):
diff --git a/tests/protocol/test_wire.py b/tests/protocol/test_wire.py
index 0c30cbfd..f26ab1b6 100644
--- a/tests/protocol/test_wire.py
+++ b/tests/protocol/test_wire.py
@@ -400,7 +400,7 @@ class TestWireProtocol(AgentTestCase):
                     # In the case Multi-config was the only feature available, 'supportedFeatures' should not be
                     # reported in the status blob as its not supported as of now.
                     # Asserting no other feature was available to report back to crp
-                    self.assertEqual(1, len(get_agent_supported_features_list_for_crp()),
+                    self.assertEqual(0, len(get_agent_supported_features_list_for_crp()),
                                      "supportedFeatures should be available if there are more features")
                     return