Repository: Azure/WALinuxAgent
Pull request #2860 (2023-06-27 01:39:57)

add time window for agent manifest download
<!-- DO NOT DELETE THIS TEMPLATE -->

## Description

Added time window for agent manifest download in the case of agent needs to update with self-update path(largestVersion)
Issue # <!-- if any -->
<!--
Please add an informative description that covers that changes made by the pull request. 
This checklist is used to make sure that common issues in a pull request are addressed.
This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
-->

---

### PR information
- [ ] The title of the PR is clear and informative.
- [ ] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
- [ ] New Unit tests were added for the changes made

### Quality of Code and Contribution Guidelines
- [ ] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/2860?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
> Merging [#2860](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/2860?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (fc364d1) into [develop](https://app.codecov.io/gh/Azure/WALinuxAgent/commit/51322761d8377d1db7af5767991b0bcd422e26e5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (5132276) will **increase** coverage by `0.02%`.
> The diff coverage is `100.00%`.

```diff
@@             Coverage Diff             @@
##           develop    #2860      +/-   ##
===========================================
+ Coverage    72.02%   72.05%   +0.02%     
===========================================
  Files          106      106              
  Lines        15974    15986      +12     
  Branches      2290     2293       +3     
===========================================
+ Hits         11506    11519      +13     
  Misses        3933     3933              
+ Partials       535      534       -1     
```


| [Impacted Files](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/2860?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/conf.py](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/2860?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9jb25mLnB5) | `80.41% <100.00%> (ø)` | |
| [azurelinuxagent/ga/agent\_update\_handler.py](https://app.codecov.io/gh/Azure/WALinuxAgent/pull/2860?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL2FnZW50X3VwZGF0ZV9oYW5kbGVyLnB5) | `90.47% <100.00%> (+0.97%)` | :arrow_up: |



Patch:
diff --git a/azurelinuxagent/common/conf.py b/azurelinuxagent/common/conf.py
index cb929e43..0e0eb7f1 100644
--- a/azurelinuxagent/common/conf.py
+++ b/azurelinuxagent/common/conf.py
@@ -129,7 +129,6 @@ __SWITCH_OPTIONS__ = {
     "ResourceDisk.EnableSwapEncryption": False,
     "AutoUpdate.Enabled": True,
     "EnableOverProvisioning": True,
-    "GAUpdates.Enabled": True,
     #
     # "Debug" options are experimental and may be removed in later
     # versions of the Agent.
@@ -137,9 +136,10 @@ __SWITCH_OPTIONS__ = {
     "Debug.CgroupLogMetrics": False,
     "Debug.CgroupDisableOnProcessCheckFailure": True,
     "Debug.CgroupDisableOnQuotaCheckFailure": True,
+    "Debug.DownloadNewAgents": True,
     "Debug.EnableAgentMemoryUsageCheck": False,
     "Debug.EnableFastTrack": True,
-    "Debug.EnableGAVersioning": False
+    "Debug.EnableGAVersioning": True
 }
 
 
@@ -503,12 +503,15 @@ def get_monitor_network_configuration_changes(conf=__conf__):
     return conf.get_switch("Monitor.NetworkConfigurationChanges", False)
 
 
-def get_ga_updates_enabled(conf=__conf__):
+def get_download_new_agents(conf=__conf__):
     """
-    If True, the agent go through update logic to look for new agents otherwise it will stop agent updates.
-    NOTE: This option is needed in e2e tests to control agent updates.
+    If True, the agent go through update logic to look for new agents to download otherwise it will stop agent updates.
+    NOTE: AutoUpdate.Enabled controls whether the Agent downloads new update and also whether any downloaded updates are started or not, while DownloadNewAgents controls only the former.
+    AutoUpdate.Enabled == false -> Agent preinstalled on the image will process extensions and will not update (regardless of DownloadNewAgents flag)
+    AutoUpdate.Enabled == true and DownloadNewAgents == true, any update already downloaded will be started, and agent look for future updates
+    AutoUpdate.Enabled == true and DownloadNewAgents == false, any update already downloaded will be started, but the agent will not look for future updates
     """
-    return conf.get_switch("GAUpdates.Enabled", True)
+    return conf.get_switch("Debug.DownloadNewAgents", True)
 
 
 def get_cgroup_check_period(conf=__conf__):
@@ -637,7 +640,7 @@ def get_normal_upgrade_frequency(conf=__conf__):
 
 def get_enable_ga_versioning(conf=__conf__):
     """
-    If True, the agent uses GA Versioning for auto-updating the agent vs automatically auto-updating to the highest version.
+    If True, the agent looks for rsm updates(checking requested version in GS) otherwise it will fall back to self-update and finds the highest version from PIR.
     NOTE: This option is experimental and may be removed in later versions of the Agent.
     """
     return conf.get_switch("Debug.EnableGAVersioning", True)
diff --git a/azurelinuxagent/ga/agent_update_handler.py b/azurelinuxagent/ga/agent_update_handler.py
index 8de6cfd8..47a8fa27 100644
--- a/azurelinuxagent/ga/agent_update_handler.py
+++ b/azurelinuxagent/ga/agent_update_handler.py
@@ -37,6 +37,7 @@ class AgentUpdateHandlerUpdateState(object):
         self.last_attempted_requested_version_update_time = datetime.datetime.min
         self.last_attempted_hotfix_update_time = datetime.datetime.min
         self.last_attempted_normal_update_time = datetime.datetime.min
+        self.last_attempted_manifest_download_time = datetime.datetime.min
 
 
 class AgentUpdateHandler(object):
@@ -86,6 +87,23 @@ class AgentUpdateHandler(object):
         else:
             self.update_state.last_attempted_normal_update_time = now
             self.update_state.last_attempted_hotfix_update_time = now
+            self.update_state.last_attempted_manifest_download_time = now
+
+    def __should_agent_attempt_manifest_download(self):
+        """
+        The agent should attempt to download the manifest if
+        the agent has not attempted to download the manifest in the last 1 hour
+        """
+        now = datetime.datetime.now()
+
+        if self.update_state.last_attempted_manifest_download_time != datetime.datetime.min:
+            next_attempt_time = self.update_state.last_attempted_manifest_download_time + datetime.timedelta(seconds=conf.get_autoupdate_frequency())
+        else:
+            next_attempt_time = now
+
+        if next_attempt_time > now:
+            return False
+        return True
 
     @staticmethod
     def __get_agent_upgrade_type(requested_version):
@@ -256,7 +274,7 @@ class AgentUpdateHandler(object):
     def run(self, goal_state):
         try:
             # Ignore new agents if update is disabled. The latter flag only used in e2e tests.
-            if not self._autoupdate_enabled or not conf.get_ga_updates_enabled():
+            if not self._autoupdate_enabled or not conf.get_download_new_agents():
                 return
 
             self._gs_id = goal_state.extensions_goal_state.id
@@ -265,6 +283,9 @@ class AgentUpdateHandler(object):
             agent_manifest = None  # This is to make sure fetch agent manifest once per update
             warn_msg = ""
             if requested_version is None:
+                # Do not proceed with update if self-update needs to download the manifest again with in an hour
+                if not self.__should_agent_attempt_manifest_download():
+                    return
                 if conf.get_enable_ga_versioning():  # log the warning only when ga versioning is enabled
                     warn_msg = "Missing requested version in agent family: {0} for incarnation: {1}, fallback to largest version update".format(self._ga_family, self._gs_id)
                     GAUpdateReportState.report_error_msg = warn_msg


Test patch:
diff --git a/tests/ga/test_agent_update_handler.py b/tests/ga/test_agent_update_handler.py
index 49b0dc76..62dfd648 100644
--- a/tests/ga/test_agent_update_handler.py
+++ b/tests/ga/test_agent_update_handler.py
@@ -28,7 +28,7 @@ class TestAgentUpdate(UpdateTestCase):
         clear_singleton_instances(ProtocolUtil)
 
     @contextlib.contextmanager
-    def __get_agent_update_handler(self, test_data=None, autoupdate_frequency=0.001, autoupdate_enabled=True):
+    def __get_agent_update_handler(self, test_data=None, autoupdate_frequency=0.001, autoupdate_enabled=True, protocol_get_error=False):
         # Default to DATA_FILE of test_data parameter raises the pylint warning
         # W0102: Dangerous default value DATA_FILE (builtins.dict) as argument (dangerous-default-value)
         test_data = DATA_FILE if test_data is None else test_data
@@ -37,9 +37,12 @@ class TestAgentUpdate(UpdateTestCase):
 
             def get_handler(url, **kwargs):
                 if HttpRequestPredicates.is_agent_package_request(url):
-                    agent_pkg = load_bin_data(self._get_agent_file_name(), self._agent_zip_dir)
-                    protocol.mock_wire_data.call_counts['agentArtifact'] += 1
-                    return MockHttpResponse(status=httpclient.OK, body=agent_pkg)
+                    if not protocol_get_error:
+                        agent_pkg = load_bin_data(self._get_agent_file_name(), self._agent_zip_dir)
+                        return MockHttpResponse(status=httpclient.OK, body=agent_pkg)
+                    else:
+                        return MockHttpResponse(status=httpclient.SERVICE_UNAVAILABLE)
+
                 return protocol.mock_wire_data.mock_http_get(url, **kwargs)
 
             def put_handler(url, *args, **_):
@@ -59,6 +62,7 @@ class TestAgentUpdate(UpdateTestCase):
                             agent_update_handler._protocol = protocol
                             yield agent_update_handler, mock_telemetry
 
+
     def __assert_agent_directories_available(self, versions):
         for version in versions:
             self.assertTrue(os.path.exists(self.agent_dir(version)), "Agent directory {0} not found".format(version))
@@ -174,6 +178,33 @@ class TestAgentUpdate(UpdateTestCase):
                 self.__assert_agent_directories_exist_and_others_dont_exist(versions=[str(CURRENT_VERSION), "99999.0.0.0"])
                 self.assertIn("Agent update found, exiting current process", ustr(context.exception.reason))
 
+    def test_it_should_not_download_manifest_again_if_last_attempted_download_time_not_elapsed(self):
+        self.prepare_agents(count=1)
+        data_file = DATA_FILE.copy()
+        data_file['ext_conf'] = "wire/ext_conf.xml"
+        with self.__get_agent_update_handler(test_data=data_file, autoupdate_frequency=10, protocol_get_error=True) as (agent_update_handler, _):
+            # making multiple agent update attempts
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+
+            mock_wire_data = agent_update_handler._protocol.mock_wire_data
+            self.assertEqual(1, mock_wire_data.call_counts['manifest_of_ga.xml'], "Agent manifest should not be downloaded again")
+
+    def test_it_should_download_manifest_if_last_attempted_download_time_is_elapsed(self):
+        self.prepare_agents(count=1)
+        data_file = DATA_FILE.copy()
+        data_file['ext_conf'] = "wire/ext_conf.xml"
+
+        with self.__get_agent_update_handler(test_data=data_file, autoupdate_frequency=0.00001, protocol_get_error=True) as (agent_update_handler, _):
+            # making multiple agent update attempts
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+            agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
+
+        mock_wire_data = agent_update_handler._protocol.mock_wire_data
+        self.assertEqual(3, mock_wire_data.call_counts['manifest_of_ga.xml'], "Agent manifest should be downloaded in all attempts")
+
     def test_it_should_not_agent_update_if_requested_version_is_same_as_current_version(self):
         data_file = DATA_FILE.copy()
         data_file["ext_conf"] = "wire/ext_conf_requested_version.xml"
@@ -313,24 +344,7 @@ class TestAgentUpdate(UpdateTestCase):
         data_file = DATA_FILE.copy()
         data_file["ext_conf"] = "wire/ext_conf_requested_version.xml"
 
-        @contextlib.contextmanager
-        def mock_agent_update_handler(test_data):
-            with mock_wire_protocol(test_data) as protocol:
-
-                def get_handler(url, **kwargs):
-                    if HttpRequestPredicates.is_agent_package_request(url):
-                        return MockHttpResponse(status=httpclient.SERVICE_UNAVAILABLE)
-                    return protocol.mock_wire_data.mock_http_get(url, **kwargs)
-
-                protocol.set_http_handlers(http_get_handler=get_handler)
-
-                with patch("azurelinuxagent.common.conf.get_autoupdate_enabled", return_value=True):
-                    with patch("azurelinuxagent.common.conf.get_autoupdate_frequency", return_value=0.001):
-                        with patch("azurelinuxagent.common.conf.get_autoupdate_gafamily", return_value="Prod"):
-                            agent_update_handler_local = get_agent_update_handler(protocol)
-                            yield agent_update_handler_local
-
-        with mock_agent_update_handler(test_data=data_file) as (agent_update_handler):
+        with self.__get_agent_update_handler(test_data=data_file, protocol_get_error=True) as (agent_update_handler, _):
             GAUpdateReportState.report_error_msg = ""
             agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
             vm_agent_update_status = agent_update_handler.get_vmagent_update_status()
@@ -343,23 +357,7 @@ class TestAgentUpdate(UpdateTestCase):
         data_file = DATA_FILE.copy()
         data_file['ext_conf'] = "wire/ext_conf.xml"
 
-        @contextlib.contextmanager
-        def mock_agent_update_handler(test_data):
-            with mock_wire_protocol(test_data) as protocol:
-                def get_handler(url, **kwargs):
-                    if HttpRequestPredicates.is_agent_package_request(url):
-                        return MockHttpResponse(status=httpclient.SERVICE_UNAVAILABLE)
-                    return protocol.mock_wire_data.mock_http_get(url, **kwargs)
-
-                protocol.set_http_handlers(http_get_handler=get_handler)
-
-                with patch("azurelinuxagent.common.conf.get_autoupdate_enabled", return_value=True):
-                    with patch("azurelinuxagent.common.conf.get_autoupdate_frequency", return_value=0.001):
-                        with patch("azurelinuxagent.common.conf.get_autoupdate_gafamily", return_value="Prod"):
-                            agent_update_handler_local = get_agent_update_handler(protocol)
-                            yield agent_update_handler_local
-
-        with mock_agent_update_handler(test_data=data_file) as (agent_update_handler):
+        with self.__get_agent_update_handler(test_data=data_file, protocol_get_error=True) as (agent_update_handler, _):
             GAUpdateReportState.report_error_msg = ""
             agent_update_handler.run(agent_update_handler._protocol.get_goal_state())
             vm_agent_update_status = agent_update_handler.get_vmagent_update_status()
diff --git a/tests/test_agent.py b/tests/test_agent.py
index f5e91405..abbc090b 100644
--- a/tests/test_agent.py
+++ b/tests/test_agent.py
@@ -40,9 +40,10 @@ Debug.CgroupDisableOnQuotaCheckFailure = True
 Debug.CgroupLogMetrics = False
 Debug.CgroupMonitorExpiryTime = 2022-03-31
 Debug.CgroupMonitorExtensionName = Microsoft.Azure.Monitor.AzureMonitorLinuxAgent
+Debug.DownloadNewAgents = True
 Debug.EnableAgentMemoryUsageCheck = False
 Debug.EnableFastTrack = True
-Debug.EnableGAVersioning = False
+Debug.EnableGAVersioning = True
 Debug.EtpCollectionPeriod = 300
 Debug.FirewallRulesLogPeriod = 86400
 DetectScvmmEnv = False
@@ -51,7 +52,6 @@ Extension.LogDir = /var/log/azure
 Extensions.Enabled = True
 Extensions.GoalStatePeriod = 6
 Extensions.InitialGoalStatePeriod = 6
-GAUpdates.Enabled = True
 HttpProxy.Host = None
 HttpProxy.Port = None
 Lib.Dir = /var/lib/waagent
diff --git a/tests_e2e/orchestrator/scripts/install-agent b/tests_e2e/orchestrator/scripts/install-agent
index b494ac8e..00200f3a 100755
--- a/tests_e2e/orchestrator/scripts/install-agent
+++ b/tests_e2e/orchestrator/scripts/install-agent
@@ -115,7 +115,7 @@ echo "Agent's conf path: $waagent_conf_path"
 sed -i 's/AutoUpdate.Enabled=n/AutoUpdate.Enabled=y/g' "$waagent_conf_path"
 # By default GAUpdates flag set to True, so that agent go through update logic to look for new agents.
 # But in e2e tests this flag needs to be off in test version 9.9.9.9 to stop the agent updates, so that our scenarios run on 9.9.9.9.
-sed -i '$a GAUpdates.Enabled=n' "$waagent_conf_path"
+sed -i '$a Debug.DownloadNewAgents=n' "$waagent_conf_path"
 
 #
 # Restart the service