Repository: Azure/WALinuxAgent
Pull request #2610 (2022-06-13 22:12:19)

monitor system-wide memory metrics
<!-- DO NOT DELETE THIS TEMPLATE -->

## Description
Created new PeriodicOperation in Monitor thread to monitor system-wide available and used memory metrics. Those metrics reported every hour.

2022-06-13T22:03:28.923508Z INFO MonitorHandler ExtHandler Event: name=WALinuxAgent, op=METRIC, message=Metric Memory/Used MBytes [] = 449.234375, duration=0
2022-06-13T22:03:28.924656Z INFO MonitorHandler ExtHandler Event: name=WALinuxAgent, op=METRIC, message=Metric Memory/Available MBytes [] = 7212.453125, duration=0

Issue # <!-- if any -->
<!--
Please add an informative description that covers that changes made by the pull request. 
This checklist is used to make sure that common issues in a pull request are addressed.
This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
-->

---

### PR information
- [ ] The title of the PR is clear and informative.
- [ ] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
- [ ] New Unit tests were added for the changes made

### Quality of Code and Contribution Guidelines
- [ ] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) Report
> Merging [#2610](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (a4acc22) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/a560872bec103523e73ddcb5fb5705896545ffce?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) (a560872) will **increase** coverage by `0.05%`.
> The diff coverage is `100.00%`.

```diff
@@             Coverage Diff             @@
##           develop    #2610      +/-   ##
===========================================
+ Coverage    71.92%   71.97%   +0.05%     
===========================================
  Files          103      103              
  Lines        15577    15609      +32     
  Branches      2478     2482       +4     
===========================================
+ Hits         11203    11235      +32     
  Misses        3864     3864              
  Partials       510      510              
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/cgroup.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9jZ3JvdXAucHk=) | `87.86% <100.00%> (+0.11%)` | :arrow_up: |
| [azurelinuxagent/common/conf.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9jb25mLnB5) | `79.66% <100.00%> (+0.17%)` | :arrow_up: |
| [azurelinuxagent/common/osutil/default.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9vc3V0aWwvZGVmYXVsdC5weQ==) | `59.91% <100.00%> (+0.63%)` | :arrow_up: |
| [azurelinuxagent/ga/monitor.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure#diff-YXp1cmVsaW51eGFnZW50L2dhL21vbml0b3IucHk=) | `76.19% <100.00%> (+1.75%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Last update [a560872...a4acc22](https://codecov.io/gh/Azure/WALinuxAgent/pull/2610?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Azure).


Patch:
diff --git a/azurelinuxagent/common/cgroup.py b/azurelinuxagent/common/cgroup.py
index 6d6b7bde..28e65e24 100644
--- a/azurelinuxagent/common/cgroup.py
+++ b/azurelinuxagent/common/cgroup.py
@@ -80,6 +80,8 @@ class MetricsCounter(object):
     MAX_MEM_USAGE = "Max Memory Usage"
     THROTTLED_TIME = "Throttled Time"
     SWAP_MEM_USAGE = "Swap Memory Usage"
+    AVAILABLE_MEM = "Available MBytes"
+    USED_MEM = "Used MBytes"
 
 
 re_user_system_times = re.compile(r'user (\d+)\nsystem (\d+)\n')
diff --git a/azurelinuxagent/common/conf.py b/azurelinuxagent/common/conf.py
index f5caf3ce..3c6e960f 100644
--- a/azurelinuxagent/common/conf.py
+++ b/azurelinuxagent/common/conf.py
@@ -563,6 +563,7 @@ def get_cgroup_monitor_expiry_time(conf=__conf__):
     """
     return conf.get("Debug.CgroupMonitorExpiryTime", "2022-03-31")
 
+
 def get_cgroup_monitor_extension_name (conf=__conf__):
     """
     cgroups monitoring extension name
diff --git a/azurelinuxagent/common/osutil/default.py b/azurelinuxagent/common/osutil/default.py
index d2ae36e8..056c50e0 100644
--- a/azurelinuxagent/common/osutil/default.py
+++ b/azurelinuxagent/common/osutil/default.py
@@ -108,6 +108,7 @@ def get_firewall_delete_conntrack_drop_command(wait, destination):
 
 PACKET_PATTERN = "^\s*(\d+)\s+(\d+)\s+DROP\s+.*{0}[^\d]*$"  # pylint: disable=W1401
 ALL_CPUS_REGEX = re.compile('^cpu .*')
+ALL_MEMS_REGEX = re.compile('^Mem.*')
 
 _enable_firewall = True
 
@@ -1411,6 +1412,27 @@ class DefaultOSUtil(object):
                     break
         return system_cpu
 
+    @staticmethod
+    def get_used_and_available_system_memory():
+        """
+        Get the contents of free -b in bytes.
+        # free -b
+        #              total        used        free      shared  buff/cache   available
+        # Mem:     8340144128   619352064  5236809728     1499136  2483982336  7426314240
+        # Swap:             0           0           0
+
+        :return: used and available memory in megabytes
+        """
+        used_mem = available_mem = 0
+        free_cmd = ["free", "-b"]
+        memory = shellutil.run_command(free_cmd)
+        for line in memory.split("\n"):
+            if ALL_MEMS_REGEX.match(line):
+                mems = line.split()
+                used_mem = int(mems[2])
+                available_mem = int(mems[6])  # see "man free" for a description of these fields
+        return used_mem/(1024 ** 2), available_mem/(1024 ** 2)
+
     def get_nic_state(self, as_string=False):
         """
         Capture NIC state (IPv4 and IPv6 addresses plus link state).
diff --git a/azurelinuxagent/ga/monitor.py b/azurelinuxagent/ga/monitor.py
index 43ebd688..e2744bc4 100644
--- a/azurelinuxagent/ga/monitor.py
+++ b/azurelinuxagent/ga/monitor.py
@@ -22,6 +22,7 @@ import threading
 import azurelinuxagent.common.conf as conf
 import azurelinuxagent.common.logger as logger
 import azurelinuxagent.common.utils.networkutil as networkutil
+from azurelinuxagent.common.cgroup import MetricValue, MetricsCategory, MetricsCounter
 from azurelinuxagent.common.cgroupconfigurator import CGroupConfigurator
 from azurelinuxagent.common.cgroupstelemetry import CGroupsTelemetry
 from azurelinuxagent.common.errorstate import ErrorState
@@ -66,6 +67,27 @@ class PollResourceUsage(PeriodicOperation):
         CGroupConfigurator.get_instance().check_cgroups(tracked_metrics)
 
 
+class PollSystemWideResourceUsage(PeriodicOperation):
+    def __init__(self):
+        super(PollSystemWideResourceUsage, self).__init__(datetime.timedelta(hours=1))
+        self.__log_metrics = conf.get_cgroup_log_metrics()
+        self.osutil = get_osutil()
+
+    def poll_system_memory_metrics(self):
+        used_mem, available_mem = self.osutil.get_used_and_available_system_memory()
+        return [
+            MetricValue(MetricsCategory.MEMORY_CATEGORY, MetricsCounter.USED_MEM, "",
+                        used_mem),
+            MetricValue(MetricsCategory.MEMORY_CATEGORY, MetricsCounter.AVAILABLE_MEM, "",
+                        available_mem)
+        ]
+
+    def _operation(self):
+        metrics = self.poll_system_memory_metrics()
+        for metric in metrics:
+            report_metric(metric.category, metric.counter, metric.instance, metric.value, log_event=self.__log_metrics)
+
+
 class ResetPeriodicLogMessages(PeriodicOperation):
     """
     Periodic operation to clean up the hash-tables maintained by the loggers. For reference, please check
@@ -274,6 +296,7 @@ class MonitorHandler(ThreadHandlerInterface):
                 ResetPeriodicLogMessages(),
                 ReportNetworkErrors(),
                 PollResourceUsage(),
+                PollSystemWideResourceUsage(),
                 SendHostPluginHeartbeat(protocol, health_service),
                 SendImdsHeartbeat(protocol_util, health_service)
             ]


Test patch:
diff --git a/tests/common/osutil/test_default.py b/tests/common/osutil/test_default.py
index 2817a634..ac551024 100644
--- a/tests/common/osutil/test_default.py
+++ b/tests/common/osutil/test_default.py
@@ -963,6 +963,28 @@ Match host 192.168.1.2\n\
         as_string = osutil.DefaultOSUtil().get_nic_state(as_string=True)
         self.assertNotEqual(as_string, '')
 
+    def test_get_used_and_available_system_memory(self):
+        memory_table = "\
+              total        used        free      shared  buff/cache   available \n\
+Mem:     8340144128   619352064  5236809728     1499136  2483982336  7426314240   \n\
+Swap:             0           0           0   \n"
+        with patch.object(shellutil, 'run_command', return_value=memory_table):
+            used_mem, available_mem = osutil.DefaultOSUtil().get_used_and_available_system_memory()
+
+        self.assertEqual(used_mem, 619352064/(1024**2), "The value didn't match")
+        self.assertEqual(available_mem, 7426314240/(1024**2), "The value didn't match")
+
+    def test_get_used_and_available_system_memory_error(self):
+        msg = 'message'
+        exception = shellutil.CommandError("free -d", 1, "", msg)
+
+        with patch.object(shellutil, 'run_command',
+                          side_effect=exception) as patch_run:
+            with self.assertRaises(shellutil.CommandError) as context_manager:
+                osutil.DefaultOSUtil().get_used_and_available_system_memory()
+            self.assertEqual(patch_run.call_count, 1)
+            self.assertEqual(context_manager.exception.returncode, 1)
+
     def test_get_dhcp_pid_should_return_a_list_of_pids(self):
         osutil_get_dhcp_pid_should_return_a_list_of_pids(self, osutil.DefaultOSUtil())
 
diff --git a/tests/ga/test_monitor.py b/tests/ga/test_monitor.py
index 9543f528..d5700bc9 100644
--- a/tests/ga/test_monitor.py
+++ b/tests/ga/test_monitor.py
@@ -29,7 +29,7 @@ from azurelinuxagent.common.protocol.util import ProtocolUtil
 from azurelinuxagent.common.protocol.wire import WireProtocol
 from azurelinuxagent.ga.monitor import get_monitor_handler, PeriodicOperation, SendImdsHeartbeat, \
     ResetPeriodicLogMessages, SendHostPluginHeartbeat, PollResourceUsage, \
-    ReportNetworkErrors, ReportNetworkConfigurationChanges
+    ReportNetworkErrors, ReportNetworkConfigurationChanges, PollSystemWideResourceUsage
 from tests.protocol.mocks import mock_wire_protocol, MockHttpResponse
 from tests.protocol.HttpRequestPredicates import HttpRequestPredicates
 from tests.protocol.mockwiredata import DATA_FILE
@@ -74,6 +74,7 @@ class MonitorHandlerTestCase(AgentTestCase):
 
                                 expected_operations = [
                                     PollResourceUsage.__name__,
+                                    PollSystemWideResourceUsage.__name__,
                                     ReportNetworkErrors.__name__,
                                     ResetPeriodicLogMessages.__name__,
                                     SendHostPluginHeartbeat.__name__,
@@ -252,3 +253,24 @@ class TestExtensionMetricsDataTelemetry(AgentTestCase):
         self.assertEqual(0, patch_periodic_warn.call_count)
         self.assertEqual(0, patch_add_metric.call_count)  # No metrics should be sent.
 
+
+class TestPollSystemWideResourceUsage(AgentTestCase):
+
+    @patch('azurelinuxagent.common.event.EventLogger.add_metric')
+    @patch("azurelinuxagent.common.osutil.default.DefaultOSUtil.get_used_and_available_system_memory")
+    def test_send_system_memory_metrics(self, path_get_system_memory, patch_add_metric, *args): # pylint: disable=unused-argument
+        path_get_system_memory.return_value = (234.45, 123.45)
+        PollSystemWideResourceUsage().run()
+
+        self.assertEqual(1, path_get_system_memory.call_count)
+        self.assertEqual(2, patch_add_metric.call_count)  # 2 metrics being sent.
+
+    @patch('azurelinuxagent.common.event.EventLogger.add_metric')
+    @patch("azurelinuxagent.ga.monitor.PollSystemWideResourceUsage.poll_system_memory_metrics")
+    def test_send_system_memory_metrics_empty(self, path_poll_system_memory_metrics, patch_add_metric, # pylint: disable=unused-argument
+                                        *args):
+        path_poll_system_memory_metrics.return_value = []
+        PollSystemWideResourceUsage().run()
+
+        self.assertEqual(1, path_poll_system_memory_metrics.call_count)
+        self.assertEqual(0, patch_add_metric.call_count)  # Zero metrics being sent.
\ No newline at end of file