Repository: Azure/WALinuxAgent
Pull request #1891 (2020-05-21 00:07:01)

Expose periods of environment thread in waagent.conf
Added parameters in waagent.conf to control how often operations in the environment thread are performed.

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=h1) Report
> Merging [#1891](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=desc) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/1268fdce6bf02b464fc87395c4c712a2fd7d5955&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `61.44%`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/graphs/tree.svg?width=650&height=150&src=pr&token=ae90E2KYCw)](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1891      +/-   ##
===========================================
- Coverage    69.20%   69.18%   -0.02%     
===========================================
  Files           83       83              
  Lines        11445    11468      +23     
  Branches      1589     1592       +3     
===========================================
+ Hits          7920     7934      +14     
- Misses        3202     3208       +6     
- Partials       323      326       +3     
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/ga/env.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL2Vudi5weQ==) | `50.86% <27.77%> (-0.54%)` | :arrow_down: |
| [azurelinuxagent/common/conf.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9jb25mLnB5) | `79.89% <78.57%> (+1.65%)` | :arrow_up: |
| [azurelinuxagent/ga/monitor.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL21vbml0b3IucHk=) | `71.62% <80.00%> (-2.91%)` | :arrow_down: |
| [azurelinuxagent/ga/periodic\_operation.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL3BlcmlvZGljX29wZXJhdGlvbi5weQ==) | `97.05% <100.00%> (+1.22%)` | :arrow_up: |
| [azurelinuxagent/ga/update.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL3VwZGF0ZS5weQ==) | `88.64% <100.00%> (-0.02%)` | :arrow_down: |
| [azurelinuxagent/pa/provision/default.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L3BhL3Byb3Zpc2lvbi9kZWZhdWx0LnB5) | `74.24% <0.00%> (-1.02%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=footer). Last update [1268fdc...580d9e4](https://codecov.io/gh/Azure/WALinuxAgent/pull/1891?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/README.md b/README.md
index dea90f41..1749efdc 100644
--- a/README.md
+++ b/README.md
@@ -183,6 +183,8 @@ A sample configuration file is shown below:
 
 ```yml
 Extensions.Enabled=y
+Extensions.GoalStatePeriod=6
+Extensions.GoalStateHistoryCleanupPeriod=86400
 Provisioning.Agent=auto
 Provisioning.DeleteRootPassword=n
 Provisioning.RegenerateSshHostKeyPair=y
@@ -236,6 +238,28 @@ without the agent. In order to do that, the `provisionVMAgent` flag must be set
 provisioning time, via whichever API is being used. We will provide more details on
 this on our wiki when it is generally available. 
 
+#### __Extensions.GoalStatePeriod__
+
+_Type: Integer_  
+_Default: 6_
+
+How often to poll for new goal states (in seconds) and report the status of the VM
+and extensions. Goal states describe the desired state of the extensions on the VM.
+
+_Note_: setting up this parameter to more than a few minutes can make the state of
+the VM be reported as unresponsive/unavailable on the Azure portal. Also, this 
+setting affects how fast the agent starts executing extensions. 
+
+#### __Extensions.GoalStateHistoryCleanupPeriod__
+
+_Type: Integer_  
+_Default: 86400 (24 hours)_
+
+How often to clean up the history folder of the agent. The agent keeps past goal
+states on this folder, each goal state represented with a set of small files. The
+history is useful to debug issues in the agent or extensions.
+ 
+
 #### __Provisioning.Agent__
 
 _Type: String_
@@ -259,7 +283,22 @@ _Note_: This configuration option has been removed and has no effect. waagent
 now auto-detects cloud-init as a provisioning agent (with an option to override
 with `Provisioning.Agent`).
 
-#### __Provisioning.UseCloudInit__ (*removed in 2.2.45*)
+#### __Provisioning.MonitorHostName__
+
+_Type: Boolean_ 
+_Default: n_
+
+Monitor host name changes and publish changes via DHCP requests.
+
+#### __Provisioning.MonitorHostNamePeriod__
+
+_Type: Integer_ 
+_Default: 30_
+
+How often to monitor host name changes (in seconds). This setting is ignored if
+MonitorHostName is not set.
+
+#### __Provisioning.UseCloudInit__
 
 _Type: Boolean_ 
 _Default: n_
@@ -440,6 +479,14 @@ OpenSSL commands. This signals OpenSSL to use any installed FIPS-compliant libra
 Note that the agent itself has no FIPS-specific code. _If no FIPS-compliant certificates are
 installed, then enabling this option will cause all OpenSSL commands to fail._
 
+#### __OS.MonitorDhcpClientRestartPeriod__
+
+_Type: Integer_
+_Default: 30_
+
+The agent monitor restarts of the DHCP client and restores network rules when it happens. This
+setting determines how often (in seconds) to monitor for restarts.
+
 #### __OS.RootDeviceScsiTimeout__
 
 _Type: Integer_  
@@ -448,6 +495,14 @@ _Default: 300_
 This configures the SCSI timeout in seconds on the root device. If not set, the
 system defaults are used.
 
+#### __OS.RootDeviceScsiTimeoutPeriod__
+
+_Type: Integer_  
+_Default: 30_
+
+How often to set the SCSI timeout on the root device (in seconds). This setting is
+ignored if RootDeviceScsiTimeout is not set.
+
 #### __OS.OpensslPath__
 
 _Type: String_  
@@ -456,6 +511,13 @@ _Default: None_
 This can be used to specify an alternate path for the openssl binary to use for
 cryptographic operations.
 
+#### __OS.RemovePersistentNetRulesPeriod__
+_Type: Integer_  
+_Default: 30_
+
+How often to remove the udev rules for persistent network interface names (75-persistent-net-generator.rules
+and /etc/udev/rules.d/70-persistent-net.rules) (in seconds) 
+
 #### __OS.SshClientAliveInterval__
 
 _Type: Integer_  
diff --git a/azurelinuxagent/common/conf.py b/azurelinuxagent/common/conf.py
index bfc61f02..7e556a84 100644
--- a/azurelinuxagent/common/conf.py
+++ b/azurelinuxagent/common/conf.py
@@ -138,7 +138,14 @@ __STRING_OPTIONS__ = {
 
 
 __INTEGER_OPTIONS__ = {
+    "Extensions.GoalStatePeriod": 6,
+    "Extensions.GoalStateHistoryCleanupPeriod": 86400,
+    "OS.EnableFirewallPeriod": 30,
+    "OS.RemovePersistentNetRulesPeriod": 30,
+    "OS.RootDeviceScsiTimeoutPeriod": 30,
+    "OS.MonitorDhcpClientRestartPeriod": 30,
     "OS.SshClientAliveInterval": 180,
+    "Provisioning.MonitorHostNamePeriod": 30,
     "Provisioning.PasswordCryptSaltLength": 10,
     "HttpProxy.Port": None,
     "ResourceDisk.SwapSizeMB": 0,
@@ -160,10 +167,40 @@ def get_configuration(conf=__conf__):
     return options
 
 
+def get_default_value(option):
+    if option in __STRING_OPTIONS__:
+        return __STRING_OPTIONS__[option]
+    raise ValueError("{0} is not a valid configuration parameter.".format(option))
+
+
+def get_int_default_value(option):
+    if option in __INTEGER_OPTIONS__:
+        return int(__INTEGER_OPTIONS__[option])
+    raise ValueError("{0} is not a valid configuration parameter.".format(option))
+
+
+def get_switch_default_value(option):
+    if option in __SWITCH_OPTIONS__:
+        return __SWITCH_OPTIONS__[option]
+    raise ValueError("{0} is not a valid configuration parameter.".format(option))
+
+
 def enable_firewall(conf=__conf__):
     return conf.get_switch("OS.EnableFirewall", False)
 
 
+def get_enable_firewall_period(conf=__conf__):
+    return conf.get_int("OS.EnableFirewallPeriod", 30)
+
+
+def get_remove_persistent_net_rules_period(conf=__conf__):
+    return conf.get_int("OS.RemovePersistentNetRulesPeriod", 30)
+
+
+def get_monitor_dhcp_client_restart_period(conf=__conf__):
+    return conf.get_int("OS.MonitorDhcpClientRestartPeriod", 30)
+
+
 def enable_rdma(conf=__conf__):
     return conf.get_switch("OS.EnableRDMA", False) or \
            conf.get_switch("OS.UpdateRdmaDriver", False) or \
@@ -256,6 +293,10 @@ def get_root_device_scsi_timeout(conf=__conf__):
     return conf.get("OS.RootDeviceScsiTimeout", None)
 
 
+def get_root_device_scsi_timeout_period(conf=__conf__):
+    return conf.get_int("OS.RootDeviceScsiTimeoutPeriod", 30)
+
+
 def get_ssh_host_keypair_type(conf=__conf__):
     keypair_type = conf.get("Provisioning.SshHostKeyPairType", "rsa")
     if keypair_type == "auto":
@@ -275,6 +316,14 @@ def get_extensions_enabled(conf=__conf__):
     return conf.get_switch("Extensions.Enabled", True)
 
 
+def get_goal_state_period(conf=__conf__):
+    return conf.get_int("Extensions.GoalStatePeriod", 6)
+
+
+def get_goal_state_history_cleanup_period(conf=__conf__):
+    return conf.get_int("Extensions.GoalStateHistoryCleanupPeriod", 86400)
+
+
 def get_allow_reset_sys_user(conf=__conf__):
     return conf.get_switch("Provisioning.AllowResetSysUser", False)
 
@@ -321,6 +370,8 @@ def get_password_crypt_salt_len(conf=__conf__):
 def get_monitor_hostname(conf=__conf__):
     return conf.get_switch("Provisioning.MonitorHostName", False)
 
+def get_monitor_hostname_period(conf=__conf__):
+    return conf.get_int("Provisioning.MonitorHostNamePeriod", 30)
 
 def get_httpproxy_host(conf=__conf__):
     return conf.get("HttpProxy.Host", None)
diff --git a/azurelinuxagent/common/event.py b/azurelinuxagent/common/event.py
index 05a46551..6d7bd384 100644
--- a/azurelinuxagent/common/event.py
+++ b/azurelinuxagent/common/event.py
@@ -71,9 +71,10 @@ class WALAEventOperation:
     AgentEnabled = "AgentEnabled"
     ArtifactsProfileBlob = "ArtifactsProfileBlob"
     AutoUpdate = "AutoUpdate"
-    CustomData = "CustomData"
     CGroupsCleanUp = "CGroupsCleanUp"
     CGroupsLimitsCrossed = "CGroupsLimitsCrossed"
+    ConfigurationChange = "ConfigurationChange"
+    CustomData = "CustomData"
     Deploy = "Deploy"
     Disable = "Disable"
     Downgrade = "Downgrade"
diff --git a/azurelinuxagent/ga/env.py b/azurelinuxagent/ga/env.py
index 9ea16155..8c899552 100644
--- a/azurelinuxagent/ga/env.py
+++ b/azurelinuxagent/ga/env.py
@@ -20,9 +20,7 @@
 import re
 import os
 import socket
-import time
 import threading
-import datetime
 
 import azurelinuxagent.common.conf as conf
 import azurelinuxagent.common.logger as logger
@@ -34,6 +32,7 @@ from azurelinuxagent.common.osutil import get_osutil
 from azurelinuxagent.common.protocol.util import get_protocol_util
 from azurelinuxagent.common.utils.archive import StateArchiver
 from azurelinuxagent.common.version import AGENT_NAME, CURRENT_VERSION
+from azurelinuxagent.ga.periodic_operation import PeriodicOperation
 
 CACHE_PATTERNS = [
     re.compile("^(.*)\.(\d+)\.(agentsManifest)$", re.IGNORECASE),
@@ -43,9 +42,6 @@ CACHE_PATTERNS = [
 
 MAXIMUM_CACHED_FILES = 50
 
-ARCHIVE_INTERVAL = datetime.timedelta(hours=24)
-
-
 def get_env_handler():
     return EnvHandler()
 
@@ -62,13 +58,26 @@ class EnvHandler(object):
         self.osutil = get_osutil()
         self.dhcp_handler = get_dhcp_handler()
         self.protocol_util = None
+        self._protocol = None
         self.stopped = True
         self.hostname = None
         self.dhcp_id_list = []
         self.server_thread = None
         self.dhcp_warning_enabled = True
-        self.last_archive = None
         self.archiver = StateArchiver(conf.get_lib_dir())
+        self._reset_firewall_rules = False
+
+        self._periodic_operations = [
+            PeriodicOperation("_remove_persistent_net_rules", self._remove_persistent_net_rules_period, conf.get_remove_persistent_net_rules_period()),
+            PeriodicOperation("_monitor_dhcp_client_restart", self._monitor_dhcp_client_restart, conf.get_monitor_dhcp_client_restart_period()),
+            PeriodicOperation("_cleanup_goal_state_history", self._cleanup_goal_state_history, conf.get_goal_state_history_cleanup_period())
+        ]
+        if conf.enable_firewall():
+            self._periodic_operations.append(PeriodicOperation("_enable_firewall", self._enable_firewall, conf.get_enable_firewall_period()))
+        if conf.get_root_device_scsi_timeout() is not None:
+            self._periodic_operations.append(PeriodicOperation("_set_root_device_scsi_timeout", self._set_root_device_scsi_timeout, conf.get_root_device_scsi_timeout_period()))
+        if conf.get_monitor_hostname():
+            self._periodic_operations.append(PeriodicOperation("_monitor_hostname", self._monitor_hostname_changes, conf.get_monitor_hostname_period()))
 
     def run(self):
         if not self.stopped:
@@ -92,56 +101,50 @@ class EnvHandler(object):
         self.server_thread.start()
 
     def monitor(self):
-        """
-        Monitor firewall rules
-        Monitor dhcp client pid and hostname.
-        If dhcp client process re-start has occurred, reset routes.
-        Purge unnecessary files from disk cache.
-        """
-
-        # The initialization of ProtocolUtil for the Environment thread should be done within the thread itself rather
-        # than initializing it in the ExtHandler thread. This is done to avoid any concurrency issues as each
-        # thread would now have its own ProtocolUtil object as per the SingletonPerThread model.
-        self.protocol_util = get_protocol_util()
-        protocol = self.protocol_util.get_protocol()
-        reset_firewall_fules = False
-        while not self.stopped:
-            self.osutil.remove_rules_files()
-
-            if conf.enable_firewall():
-                # If the rules ever change we must reset all rules and start over again.
-                #
-                # There was a rule change at 2.2.26, which started dropping non-root traffic
-                # to WireServer.  The previous rules allowed traffic.  Having both rules in
-                # place negated the fix in 2.2.26.
-                if not reset_firewall_fules:
-                    self.osutil.remove_firewall(dst_ip=protocol.get_endpoint(), uid=os.getuid())
-                    reset_firewall_fules = True
-
-                success = self.osutil.enable_firewall(dst_ip=protocol.get_endpoint(), uid=os.getuid())
-
-                add_periodic(
-                    logger.EVERY_HOUR,
-                    AGENT_NAME,
-                    version=CURRENT_VERSION,
-                    op=WALAEventOperation.Firewall,
-                    is_success=success,
-                    log_event=False)
-
-            timeout = conf.get_root_device_scsi_timeout()
-            if timeout is not None:
-                self.osutil.set_scsi_disks_timeout(timeout)
-
-            if conf.get_monitor_hostname():
-                self.handle_hostname_update()
-
-            self.handle_dhclient_restart()
-
-            self.archive_history()
-
-            time.sleep(5)
-
-    def handle_hostname_update(self):
+        try:
+            # The initialization of ProtocolUtil for the Environment thread should be done within the thread itself rather
+            # than initializing it in the ExtHandler thread. This is done to avoid any concurrency issues as each
+            # thread would now have its own ProtocolUtil object as per the SingletonPerThread model.
+            self.protocol_util = get_protocol_util()
+            self._protocol = self.protocol_util.get_protocol()
+            while not self.stopped:
+                try:
+                    for op in self._periodic_operations:
+                        op.run()
+                except Exception as e:
+                    logger.error("An error occurred in the environment thread main loop; will skip the current iteration.\n{0}", ustr(e))
+                finally:
+                    PeriodicOperation.sleep_until_next_operation(self._periodic_operations)
+        except Exception as e:
+            logger.error("An error occurred in the environment thread; will exit the thread.\n{0}", ustr(e))
+
+    def _remove_persistent_net_rules_period(self):
+        self.osutil.remove_rules_files()
+
+    def _enable_firewall(self):
+        # If the rules ever change we must reset all rules and start over again.
+        #
+        # There was a rule change at 2.2.26, which started dropping non-root traffic
+        # to WireServer.  The previous rules allowed traffic.  Having both rules in
+        # place negated the fix in 2.2.26.
+        if not self._reset_firewall_rules:
+            self.osutil.remove_firewall(dst_ip=self._protocol.get_endpoint(), uid=os.getuid())
+            self._reset_firewall_rules = True
+
+        success = self.osutil.enable_firewall(dst_ip=self._protocol.get_endpoint(), uid=os.getuid())
+
+        add_periodic(
+            logger.EVERY_HOUR,
+            AGENT_NAME,
+            version=CURRENT_VERSION,
+            op=WALAEventOperation.Firewall,
+            is_success=success,
+            log_event=False)
+
+    def _set_root_device_scsi_timeout(self):
+        self.osutil.set_scsi_disks_timeout(conf.get_root_device_scsi_timeout())
+
+    def _monitor_hostname_changes(self):
         curr_hostname = socket.gethostname()
         if curr_hostname != self.hostname:
             logger.info("EnvMonitor: Detected hostname change: {0} -> {1}",
@@ -169,6 +172,9 @@ class EnvHandler(object):
 
         return pid
 
+    def _monitor_dhcp_client_restart(self):
+        self.handle_dhclient_restart()
+
     def handle_dhclient_restart(self):
         if len(self.dhcp_id_list) == 0:
             self.dhcp_id_list = self.get_dhcp_client_pid()
@@ -183,16 +189,10 @@ class EnvHandler(object):
             self.dhcp_handler.conf_routes()
             self.dhcp_id_list = new_pid
 
-    def archive_history(self):
+    def _cleanup_goal_state_history(self):
         """
-        Purge history if we have exceed the maximum count.
-        Create a .zip of the history that has been preserved.
+        Purge history and create a .zip of the history that has been preserved.
         """
-        if self.last_archive is not None \
-                and datetime.datetime.utcnow() < \
-                self.last_archive + ARCHIVE_INTERVAL:
-            return
-
         self.archiver.purge()
         self.archiver.archive()
 
diff --git a/azurelinuxagent/ga/monitor.py b/azurelinuxagent/ga/monitor.py
index ad2a9650..e230063c 100644
--- a/azurelinuxagent/ga/monitor.py
+++ b/azurelinuxagent/ga/monitor.py
@@ -74,13 +74,15 @@ class MonitorHandler(object):
         self.imds_client = None
 
         self.event_thread = None
-        self._reset_loggers_op = PeriodicOperation("reset_loggers", self.reset_loggers, self.RESET_LOGGERS_PERIOD)
-        self._collect_and_send_events_op = PeriodicOperation("collect_and_send_events", self.collect_and_send_events, self.EVENT_COLLECTION_PERIOD)
-        self._send_telemetry_heartbeat_op = PeriodicOperation("send_telemetry_heartbeat", self.send_telemetry_heartbeat, self.TELEMETRY_HEARTBEAT_PERIOD)
-        self._poll_telemetry_metrics_op = PeriodicOperation("poll_telemetry_metrics usage", self.poll_telemetry_metrics, self.CGROUP_TELEMETRY_POLLING_PERIOD)
-        self._send_host_plugin_heartbeat_op = PeriodicOperation("send_host_plugin_heartbeat", self.send_host_plugin_heartbeat, self.HOST_PLUGIN_HEARTBEAT_PERIOD)
-        self._send_imds_heartbeat_op = PeriodicOperation("send_imds_heartbeat", self.send_imds_heartbeat, self.IMDS_HEARTBEAT_PERIOD)
-        self._log_altered_network_configuration_op = PeriodicOperation("log_altered_network_configuration", self.log_altered_network_configuration, self.LOG_NETWORK_CONFIGURATION_PERIOD)
+        self._periodic_operations = [
+            PeriodicOperation("reset_loggers", self.reset_loggers, self.RESET_LOGGERS_PERIOD),
+            PeriodicOperation("collect_and_send_events", self.collect_and_send_events, self.EVENT_COLLECTION_PERIOD),
+            PeriodicOperation("send_telemetry_heartbeat", self.send_telemetry_heartbeat, self.TELEMETRY_HEARTBEAT_PERIOD),
+            PeriodicOperation("poll_telemetry_metrics usage", self.poll_telemetry_metrics, self.CGROUP_TELEMETRY_POLLING_PERIOD),
+            PeriodicOperation("send_host_plugin_heartbeat", self.send_host_plugin_heartbeat, self.HOST_PLUGIN_HEARTBEAT_PERIOD),
+            PeriodicOperation("send_imds_heartbeat", self.send_imds_heartbeat, self.IMDS_HEARTBEAT_PERIOD),
+            PeriodicOperation("log_altered_network_configuration", self.log_altered_network_configuration, self.LOG_NETWORK_CONFIGURATION_PERIOD),
+        ]
         self.protocol = None
         self.protocol_util = None
         self.health_service = None
@@ -137,29 +139,24 @@ class MonitorHandler(object):
             self.protocol.report_event(event_list)
 
     def daemon(self, init_data=False):
+        try:
+            if init_data:
+                self.init_protocols()
+                self.init_imds_client()
+
+            while not self.stopped():
+                try:
+                    self.protocol.update_host_plugin_from_goal_state()
 
-        if init_data:
-            self.init_protocols()
-            self.init_imds_client()
-
-        min_delta = min(MonitorHandler.TELEMETRY_HEARTBEAT_PERIOD,
-                        MonitorHandler.CGROUP_TELEMETRY_POLLING_PERIOD,
-                        MonitorHandler.EVENT_COLLECTION_PERIOD,
-                        MonitorHandler.HOST_PLUGIN_HEARTBEAT_PERIOD,
-                        MonitorHandler.IMDS_HEARTBEAT_PERIOD).seconds
-        while not self.stopped():
-            try:
-                self.protocol.update_host_plugin_from_goal_state()
-                self._send_telemetry_heartbeat_op.run()
-                self._poll_telemetry_metrics_op.run()
-                self._collect_and_send_events_op.run()
-                self._send_host_plugin_heartbeat_op.run()
-                self._send_imds_heartbeat_op.run()
-                self._log_altered_network_configuration_op.run()
-                self._reset_loggers_op.run()
-            except Exception as e:
-                logger.warn("An error occurred in the monitor thread main loop; will skip the current iteration.\n{0}", ustr(e))
-            time.sleep(min_delta)
+                    for op in self._periodic_operations:
+                        op.run()
+
+                except Exception as e:
+                    logger.error("An error occurred in the monitor thread main loop; will skip the current iteration.\n{0}", ustr(e))
+                finally:
+                    PeriodicOperation.sleep_until_next_operation(self._periodic_operations)
+        except Exception as e:
+            logger.error("An error occurred in the monitor thread; will exit the thread.\n{0}", ustr(e))
 
     def reset_loggers(self):
         """
diff --git a/azurelinuxagent/ga/periodic_operation.py b/azurelinuxagent/ga/periodic_operation.py
index 1135586e..5c54f1d9 100644
--- a/azurelinuxagent/ga/periodic_operation.py
+++ b/azurelinuxagent/ga/periodic_operation.py
@@ -16,6 +16,7 @@
 #
 
 import datetime
+import time
 
 from azurelinuxagent.common import logger
 from azurelinuxagent.common.future import ustr
@@ -35,18 +36,18 @@ class PeriodicOperation(object):
     def __init__(self, name, operation, period):
         self._name = name
         self._operation = operation
-        self._period = period
-        self._last_run = None
+        self._period = period if isinstance(period, datetime.timedelta) else datetime.timedelta(seconds=period)
+        self._next_run_time = datetime.datetime.utcnow()
         self._last_warning = None
         self._last_warning_time = None
 
     def run(self):
         try:
-            if self._last_run is None or datetime.datetime.utcnow() >= self._last_run + self._period:
+            if self._next_run_time <= datetime.datetime.utcnow():
                 try:
                     self._operation()
                 finally:
-                    self._last_run = datetime.datetime.utcnow()
+                    self._next_run_time = datetime.datetime.utcnow() + self._period
         except Exception as e:
             warning = "Failed to {0}: {1} --- [NOTE: Will not log the same error for the next hour]".format(self._name, ustr(e))
             if warning != self._last_warning or self._last_warning_time is None or datetime.datetime.utcnow() >= self._last_warning_time + self._LOG_WARNING_PERIOD:
@@ -54,3 +55,21 @@ class PeriodicOperation(object):
                 self._last_warning_time = datetime.datetime.utcnow()
                 self._last_warning = warning
 
+    def next_run_time(self):
+        return self._next_run_time
+
+    @staticmethod
+    def sleep_until_next_operation(operations):
+        """
+        Takes a list of operations, finds the operation that should be executed next (that with the closest next_run_time)
+        and sleeps until it is time to execute that operation.
+        """
+        next_operation_time = min([op.next_run_time() for op in operations])
+
+        sleep_timedelta = next_operation_time - datetime.datetime.utcnow()
+        # timedelta.total_seconds() is not available on Python 2.6, do the computation manually
+        sleep_seconds = ((sleep_timedelta.days * 24 * 3600 + sleep_timedelta.seconds) * 10.0 ** 6 + sleep_timedelta.microseconds) / 10.0 ** 6
+
+        if sleep_seconds > 0:
+            time.sleep(sleep_seconds)
+
diff --git a/azurelinuxagent/ga/update.py b/azurelinuxagent/ga/update.py
index fa5fdc20..60fea307 100644
--- a/azurelinuxagent/ga/update.py
+++ b/azurelinuxagent/ga/update.py
@@ -48,7 +48,6 @@ from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.osutil import get_osutil
 from azurelinuxagent.common.protocol.util import get_protocol_util
 from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol
-from azurelinuxagent.common.protocol.wire import WireProtocol
 from azurelinuxagent.common.utils.flexible_version import FlexibleVersion
 from azurelinuxagent.common.version import AGENT_NAME, AGENT_VERSION, AGENT_DIR_PATTERN, CURRENT_AGENT,\
     CURRENT_VERSION, DISTRO_NAME, DISTRO_VERSION, is_current_agent_installed, PY_VERSION_MAJOR, PY_VERSION_MINOR, \
@@ -67,9 +66,9 @@ CHILD_POLL_INTERVAL = 60
 
 MAX_FAILURE = 3 # Max failure allowed for agent before blacklisted
 
-GOAL_STATE_INTERVAL = 3
 GOAL_STATE_INTERVAL_DISABLED = 5 * 60
 
+ORPHAN_POLL_INTERVAL = 3
 ORPHAN_WAIT_INTERVAL = 15 * 60
 
 AGENT_SENTINEL_FILE = "current_version"
@@ -286,11 +285,12 @@ class UpdateHandler(object):
 
             self._ensure_no_orphans()
             self._emit_restart_event()
+            self._emit_changes_in_default_configuration()
             self._ensure_partition_assigned()
             self._ensure_readonly_files()
             self._ensure_cgroups_initialized()
 
-            goal_state_interval = GOAL_STATE_INTERVAL if conf.get_extensions_enabled() else GOAL_STATE_INTERVAL_DISABLED
+            goal_state_interval = conf.get_goal_state_period() if conf.get_extensions_enabled() else GOAL_STATE_INTERVAL_DISABLED
 
             while self.running:
                 #
@@ -422,6 +422,28 @@ class UpdateHandler(object):
 
         return
 
+    @staticmethod
+    def _emit_changes_in_default_configuration():
+        try:
+            def log_if_int_changed_from_default(name, current):
+                default = conf.get_int_default_value(name)
+                if default != current:
+                    msg = "{0} changed from its default; new value: {1}".format(name, current)
+                    logger.info(msg)
+                    add_event(AGENT_NAME, op=WALAEventOperation.ConfigurationChange, message=msg)
+
+            log_if_int_changed_from_default("Extensions.GoalStatePeriod", conf.get_goal_state_period())
+
+            if not conf.enable_firewall():
+                message = "OS.EnableFirewall is False"
+                logger.info(message)
+                add_event(AGENT_NAME, op=WALAEventOperation.ConfigurationChange, message=message)
+            else:
+                log_if_int_changed_from_default("OS.EnableFirewallPeriod", conf.get_enable_firewall_period())
+
+        except Exception as e:
+            logger.warn("Failed to log changes in configuration: {0}", ustr(e))
+
     def _ensure_no_orphans(self, orphan_wait_interval=ORPHAN_WAIT_INTERVAL):
         pid_files, ignored = self._write_pid_file()
         for pid_file in pid_files:
@@ -430,7 +452,7 @@ class UpdateHandler(object):
                 wait_interval = orphan_wait_interval
 
                 while self.osutil.check_pid_alive(pid):
-                    wait_interval -= GOAL_STATE_INTERVAL
+                    wait_interval -= ORPHAN_POLL_INTERVAL
                     if wait_interval <= 0:
                         logger.warn(
                             u"{0} forcibly terminated orphan process {1}",
@@ -443,7 +465,7 @@ class UpdateHandler(object):
                         u"{0} waiting for orphan process {1} to terminate",
                         CURRENT_AGENT,
                         pid)
-                    time.sleep(GOAL_STATE_INTERVAL)
+                    time.sleep(ORPHAN_POLL_INTERVAL)
 
                 os.remove(pid_file)
 
diff --git a/config/waagent.conf b/config/waagent.conf
index b558bb46..c5d2b1ff 100644
--- a/config/waagent.conf
+++ b/config/waagent.conf
@@ -6,6 +6,12 @@
 # backup, monitoring, or any extension handling whatsoever.
 Extensions.Enabled=y
 
+# How often (in seconds) to poll for new goal states
+Extensions.GoalStatePeriod=6
+
+# How often (in seconds) to clean up the goal state history. The default value is 24 hrs
+Extensions.GoalStateHistoryCleanupPeriod=86400
+
 # Which provisioning agent to use. Supported values are "auto" (default), "waagent",
 # "cloud-init", or "disabled".
 Provisioning.Agent=auto
@@ -23,6 +29,9 @@ Provisioning.SshHostKeyPairType=rsa
 # Monitor host name changes and publish changes via DHCP requests.
 Provisioning.MonitorHostName=y
 
+# How often (in seconds) to monitor host name changes.
+Provisioning.MonitorHostNamePeriod=30
+
 # Decode CustomData from Base64.
 Provisioning.DecodeCustomData=n
 
@@ -69,6 +78,9 @@ OS.EnableFIPS=n
 # Root device timeout in seconds.
 OS.RootDeviceScsiTimeout=300
 
+# How often (in seconds) to set the root device timeout.
+OS.RootDeviceScsiTimeoutPeriod=30
+
 # If "None", the system default version is used.
 OS.OpensslPath=None
 
@@ -124,6 +136,15 @@ OS.SshDir=/etc/ssh
 # Add firewall rules to protect access to Azure host node services
 OS.EnableFirewall=y
 
+# How often (in seconds) to check the firewall rules
+OS.EnableFirewallPeriod=30
+
+# How often (in seconds) to remove the udev rules for persistent network interface names (75-persistent-net-generator.rules and /etc/udev/rules.d/70-persistent-net.rules)
+OS.RemovePersistentNetRulesPeriod=30
+
+# How often (in seconds) to monitor for DHCP client restarts
+OS.MonitorDhcpClientRestartPeriod=30
+
 # Enforce control groups limits on the agent and extensions
 CGroups.EnforceLimits=n
 


Test patch:
diff --git a/tests/ga/test_periodic_operation.py b/tests/ga/test_periodic_operation.py
index 80a95ba7..8ed0ecce 100644
--- a/tests/ga/test_periodic_operation.py
+++ b/tests/ga/test_periodic_operation.py
@@ -21,6 +21,32 @@ from tests.tools import AgentTestCase, patch, PropertyMock
 
 
 class TestPeriodicOperation(AgentTestCase):
+    def test_it_should_take_a_timedelta_as_period(self):
+        def operation():
+            operation.run_time = datetime.datetime.utcnow()
+        operation.run_time = None
+
+        op = PeriodicOperation("test_operation", operation, period=datetime.timedelta(hours=1))
+        op.run()
+
+        expected = operation.run_time + datetime.timedelta(hours=1)
+        difference = op.next_run_time() - expected
+        self.assertTrue(difference < datetime.timedelta(seconds=1),
+            "The next run time exceeds the expected value by more than 1 second: {0} vs {1}".format(op.next_run_time(), expected))
+
+    def test_it_should_take_a_number_of_seconds_as_period(self):
+        def operation():
+            operation.run_time = datetime.datetime.utcnow()
+        operation.run_time = None
+
+        op = PeriodicOperation("test_operation", operation, period=3600)
+        op.run()
+
+        expected = operation.run_time + datetime.timedelta(hours=1)
+        difference = op.next_run_time() - expected
+        self.assertTrue(difference < datetime.timedelta(seconds=1),
+            "The next run time exceeds the expected value by more than 1 second: {0} vs {1}".format(op.next_run_time(), expected))
+
     def test_it_should_be_invoked_when_run_is_called_first_time(self):
         def operation():
             operation.invoked = True
@@ -76,7 +102,7 @@ class TestPeriodicOperation(AgentTestCase):
 
         self.assertEqual(self._get_number_of_warnings(warn_patcher), 1, "The error in the operation was should have been reported exactly once")
 
-    def test_it_should_not_multiple_warnings_when_the_period_has_elapsed(self):
+    def test_it_should_not_log_multiple_warnings_when_the_period_has_elapsed(self):
         with patch("azurelinuxagent.common.logger.warn") as warn_patcher:
             with patch("azurelinuxagent.ga.periodic_operation.PeriodicOperation._LOG_WARNING_PERIOD", new_callable=PropertyMock, return_value=datetime.timedelta(milliseconds=1)):
                 pop = PeriodicOperation("test_operation", self._operation_with_failure, period=datetime.timedelta(milliseconds=1))
@@ -98,3 +124,24 @@ class TestPeriodicOperation(AgentTestCase):
 
             self.assertEqual(self._get_number_of_warnings(warn_patcher, "WARNING 0"), 1, "The first error should have been reported exactly 1 time")
             self.assertEqual(self._get_number_of_warnings(warn_patcher, "WARNING 1"), 1, "The second error should have been reported exactly 1 time")
+
+    def test_sleep_until_next_operation_should_wait_for_the_closest_operation(self):
+        operations = [
+            PeriodicOperation("one", lambda: None, period=datetime.timedelta(seconds=60)),
+            PeriodicOperation("one", lambda: None, period=datetime.timedelta(hours=1)),
+            PeriodicOperation("one", lambda: None, period=datetime.timedelta(seconds=10)),  # closest operation
+            PeriodicOperation("one", lambda: None, period=datetime.timedelta(minutes=11)),
+            PeriodicOperation("one", lambda: None, period=datetime.timedelta(days=1))
+        ]
+        for op in operations:
+            op.run()
+
+        def mock_sleep(seconds):
+            mock_sleep.seconds = seconds
+        mock_sleep.seconds = 0
+
+        with patch("azurelinuxagent.ga.periodic_operation.time.sleep", side_effect=mock_sleep):
+            PeriodicOperation.sleep_until_next_operation(operations)
+            self.assertAlmostEqual(mock_sleep.seconds, 10, 0, "did not sleep for the expected time")
+
+
diff --git a/tests/ga/test_update.py b/tests/ga/test_update.py
index 1f0c61ab..67a11d45 100644
--- a/tests/ga/test_update.py
+++ b/tests/ga/test_update.py
@@ -7,17 +7,14 @@ import tempfile
 import unittest
 from threading import currentThread
 
-from azurelinuxagent.ga.env import EnvHandler
-
 from azurelinuxagent.common.protocol.goal_state import ExtensionsConfig
 from azurelinuxagent.common.protocol.hostplugin import *
 from azurelinuxagent.common.protocol.util import ProtocolUtil
 from azurelinuxagent.common.protocol.wire import *
 from azurelinuxagent.common.version import AGENT_PKG_GLOB, AGENT_DIR_GLOB
-from azurelinuxagent.ga.monitor import MonitorHandler
 from azurelinuxagent.ga.update import *
 from tests.tools import AgentTestCase, call, data_dir, DEFAULT, patch, load_bin_data, load_data, Mock, MagicMock, \
-    mock_sleep, clear_singleton_instances, skip_if_predicate_true
+    clear_singleton_instances
 
 NO_ERROR = {
     "last_failure": 0.0,
@@ -745,7 +742,7 @@ class TestUpdate(UpdateTestCase):
         with patch('os.kill') as mock_kill:
             calls, sleeps = self._test_ensure_no_orphans(
                 invocations=4,
-                interval=3 * GOAL_STATE_INTERVAL,
+                interval=3 * ORPHAN_POLL_INTERVAL,
                 pid_count=1)
             self.assertEqual(3, calls)
             self.assertEqual(2, sleeps)
@@ -1209,7 +1206,7 @@ class TestUpdate(UpdateTestCase):
         self._test_run_latest()
         self.assertEqual(0, mock_signal.call_count)
 
-    def _test_run(self, invocations=1, calls=[call.run()], enable_updates=False, sleep_interval=(3,)):
+    def _test_run(self, invocations=1, calls=[call.run()], enable_updates=False, sleep_interval=(6,)):
         conf.get_autoupdate_enabled = Mock(return_value=enable_updates)
 
         # Note:
diff --git a/tests/test_agent.py b/tests/test_agent.py
index dedc9905..2e89bc90 100644
--- a/tests/test_agent.py
+++ b/tests/test_agent.py
@@ -33,6 +33,8 @@ DetectScvmmEnv = False
 EnableOverProvisioning = True
 Extension.LogDir = /var/log/azure
 Extensions.Enabled = True
+Extensions.GoalStateHistoryCleanupPeriod = 86400
+Extensions.GoalStatePeriod = 6
 HttpProxy.Host = None
 HttpProxy.Port = None
 Lib.Dir = /var/lib/waagent
@@ -42,11 +44,15 @@ OS.AllowHTTP = False
 OS.CheckRdmaDriver = False
 OS.EnableFIPS = True
 OS.EnableFirewall = False
+OS.EnableFirewallPeriod = 30
 OS.EnableRDMA = False
 OS.HomeDir = /home
+OS.MonitorDhcpClientRestartPeriod = 30
 OS.OpensslPath = /usr/bin/openssl
 OS.PasswordPath = /etc/shadow
+OS.RemovePersistentNetRulesPeriod = 30
 OS.RootDeviceScsiTimeout = 300
+OS.RootDeviceScsiTimeoutPeriod = 30
 OS.SshClientAliveInterval = 42
 OS.SshDir = /notareal/path
 OS.SudoersDir = /etc/sudoers.d
@@ -58,6 +64,7 @@ Provisioning.DecodeCustomData = False
 Provisioning.DeleteRootPassword = True
 Provisioning.ExecuteCustomData = False
 Provisioning.MonitorHostName = True
+Provisioning.MonitorHostNamePeriod = 30
 Provisioning.PasswordCryptId = 6
 Provisioning.PasswordCryptSaltLength = 10
 Provisioning.RegenerateSshHostKeyPair = True
@@ -170,7 +177,7 @@ class TestAgent(AgentTestCase):
         configuration = conf.get_configuration()
         for k in sorted(configuration.keys()):
             actual_configuration.append("{0} = {1}".format(k, configuration[k]))
-        self.assertEqual(EXPECTED_CONFIGURATION, actual_configuration)
+        self.assertListEqual(EXPECTED_CONFIGURATION, actual_configuration)
 
     def test_agent_usage_message(self):
         message = usage()