Repository: Azure/WALinuxAgent
Pull request #1283 (2018-08-01 19:57:26)

Fix for test_isloopback
test_isloopback assumes there is a network interface named 'eth0'; this is not true on my system.

Instead, I am enumerating all the interfaces and running the test on all of them.


Patch:
diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
new file mode 100644
index 00000000..21a09c79
--- /dev/null
+++ b/.github/CONTRIBUTING.md
@@ -0,0 +1,77 @@
+# Contributing to Linux Guest Agent
+First, thank you for contributing to WALinuxAgent repository!
+
+## Basics
+If you would like to become an active contributor to this project, please follow the instructions provided in [Microsoft Azure Projects Contribution Guidelines](http://azure.github.io/guidelines/).
+
+## Table of Contents
+[Before starting](#before-starting)
+- [Github basics](#github-basics)
+- [Code of Conduct](#code-of-conduct)
+
+[Making Changes](#making-changes)
+- [Pull Requests](#pull-requests)
+- [Pull Request Guidelines](#pull-request-guidelines)
+    - [Cleaning up commits](#cleaning-up-commits)
+    - [General guidelines](#general-guidelines)
+    - [Testing guidelines](#testing-guidelines)
+
+## Before starting
+
+### Github basics
+
+#### GitHub workflow
+
+If you don't have experience with Git and Github, some of the terminology and process can be confusing. [Here's a guide to understanding Github](https://guides.github.com/introduction/flow/).
+
+#### Forking the Azure/Guest-Configuration-Extension repository
+
+Unless you are working with multiple contributors on the same file, we ask that you fork the repository and submit your Pull Request from there. [Here's a guide to forks in Github](https://guides.github.com/activities/forking/).
+
+### Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+## Making Changes
+
+### Pull Requests
+
+You can find all of the pull requests that have been opened in the [Pull Request](https://github.com/Azure/Guest-Configuration-Extension/pulls) section of the repository.
+
+To open your own pull request, click [here](https://github.com/Azure/WALinuxAgent/compare). When creating a pull request, keep the following in mind:
+- Make sure you are pointing to the fork and branch that your changes were made in
+- Choose the correct branch you want your pull request to be merged into
+- The pull request template that is provided **should be filled out**; this is not something that should just be deleted or ignored when the pull request is created
+    - Deleting or ignoring this template will elongate the time it takes for your pull request to be reviewed
+<!-- - The SLA for reviewing pull requests is **two business days** -->
+
+### Pull Request Guidelines
+
+A pull request template will automatically be included as a part of your PR. Please fill out the checklist as specified. Pull requests **will not be reviewed** unless they include a properly completed checklist.
+
+#### Cleaning up Commits
+
+If you are thinking about making a large change, **break up the change into small, logical, testable chunks, and organize your pull requests accordingly**.
+
+Often when a pull request is created with a large number of files changed and/or a large number of lines of code added and/or removed, GitHub will have a difficult time opening up the changes on their site. This forces the Azure Guest-Configuration-Extension team to use separate software to do a code review on the pull request.
+
+If you find yourself creating a pull request and are unable to see all the changes on GitHub, we recommend **splitting the pull request into multiple pull requests that are able to be reviewed on GitHub**.
+
+If splitting up the pull request is not an option, we recommend **creating individual commits for different parts of the pull request, which can be reviewed individually on GitHub**.
+
+For more information on cleaning up the commits in a pull request, such as how to rebase, squash, and cherry-pick, click [here](https://github.com/Azure/azure-powershell/blob/dev/documentation/cleaning-up-commits.md).
+
+#### General guidelines
+
+The following guidelines must be followed in **EVERY** pull request that is opened.
+
+- Title of the pull request is clear and informative
+- There are a small number of commits that each have an informative message
+- A description of the changes the pull request makes is included, and a reference to the issue being resolved, if the change address any
+- All files have the Microsoft copyright header
+
+#### Testing Guidelines
+
+The following guidelines must be followed in **EVERY** pull request that is opened.
+
+- Pull request includes test coverage for the included changes
\ No newline at end of file
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
new file mode 100644
index 00000000..edfa1e66
--- /dev/null
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,22 @@
+<!-- DO NOT DELETE THIS TEMPLATE -->
+
+## Description
+
+Issue # <!-- if any -->
+<!--
+Please add an informative description that covers that changes made by the pull request. 
+This checklist is used to make sure that common issues in a pull request are addressed.
+This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
+-->
+
+---
+
+### PR information
+- [ ] The title of the PR is clear and informative.
+- [ ] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
+- [ ] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
+- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
+- [ ] New Unit tests were added for the changes made and Travis.CI is passing.
+
+### Quality of Code and Contribution Guidelines
+- [ ] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).
\ No newline at end of file
diff --git a/azurelinuxagent/agent.py b/azurelinuxagent/agent.py
index e1dfc4ed..bf0a89b6 100644
--- a/azurelinuxagent/agent.py
+++ b/azurelinuxagent/agent.py
@@ -37,6 +37,7 @@ from azurelinuxagent.common.version import AGENT_NAME, AGENT_LONG_VERSION, \
                                      PY_VERSION_MAJOR, PY_VERSION_MINOR, \
                                      PY_VERSION_MICRO, GOAL_STATE_AGENT_VERSION
 from azurelinuxagent.common.osutil import get_osutil
+from azurelinuxagent.common.utils import fileutil
 
 class Agent(object):
     def __init__(self, verbose, conf_file_path=None):
@@ -73,7 +74,7 @@ class Agent(object):
             if os.path.isfile(ext_log_dir):
                 raise Exception("{0} is a file".format(ext_log_dir))
             if not os.path.isdir(ext_log_dir):
-                os.makedirs(ext_log_dir)
+                fileutil.mkdir(ext_log_dir, mode=0o755, owner="root")
         except Exception as e:
             logger.error(
                 "Exception occurred while creating extension "
diff --git a/azurelinuxagent/common/dhcp.py b/azurelinuxagent/common/dhcp.py
index 84afeccc..70a072e4 100644
--- a/azurelinuxagent/common/dhcp.py
+++ b/azurelinuxagent/common/dhcp.py
@@ -150,6 +150,15 @@ class DhcpHandler(object):
         return None
 
     def send_dhcp_req(self):
+        """
+        Check if DHCP is available
+        """
+        (dhcp_available, endpoint) =  self.osutil.is_dhcp_available()
+        if not dhcp_available:
+            logger.info("send_dhcp_req: DHCP not available")
+            self.endpoint = endpoint
+            return
+
         """
         Build dhcp request with mac addr
         Configure route to allow dhcp traffic
diff --git a/azurelinuxagent/common/event.py b/azurelinuxagent/common/event.py
index 09117acf..71723d45 100644
--- a/azurelinuxagent/common/event.py
+++ b/azurelinuxagent/common/event.py
@@ -33,7 +33,7 @@ from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.restapi import TelemetryEventParam, \
     TelemetryEvent, \
     get_properties
-from azurelinuxagent.common.utils import textutil
+from azurelinuxagent.common.utils import fileutil, textutil
 from azurelinuxagent.common.version import CURRENT_VERSION
 
 _EVENT_MSG = "Event: name={0}, op={1}, message={2}, duration={3}"
@@ -200,9 +200,7 @@ class EventLogger(object):
             logger.warn("Cannot save event -- Event reporter is not initialized.")
             return
 
-        if not os.path.exists(self.event_dir):
-            os.mkdir(self.event_dir)
-            os.chmod(self.event_dir, 0o700)
+        fileutil.mkdir(self.event_dir, mode=0o700)
 
         existing_events = os.listdir(self.event_dir)
         if len(existing_events) >= 1000:
diff --git a/azurelinuxagent/common/exception.py b/azurelinuxagent/common/exception.py
index ef026fb0..7e144616 100644
--- a/azurelinuxagent/common/exception.py
+++ b/azurelinuxagent/common/exception.py
@@ -56,8 +56,9 @@ class ExtensionError(AgentError):
     When failed to execute an extension
     """
 
-    def __init__(self, msg=None, inner=None):
+    def __init__(self, msg=None, inner=None, code=-1):
         super(ExtensionError, self).__init__(msg, inner)
+        self.code = code
 
 
 class ProvisionError(AgentError):
@@ -159,3 +160,12 @@ class ResourceGoneError(HttpError):
         if msg is None:
             msg = "Resource is gone"
         super(ResourceGoneError, self).__init__(msg, inner)
+
+
+class RemoteAccessError(AgentError):
+    """
+    Remote Access Error
+    """
+
+    def __init__(self, msg=None, inner=None):
+        super(RemoteAccessError, self).__init__(msg, inner)
diff --git a/azurelinuxagent/common/osutil/default.py b/azurelinuxagent/common/osutil/default.py
index 9667cdf2..6611424a 100644
--- a/azurelinuxagent/common/osutil/default.py
+++ b/azurelinuxagent/common/osutil/default.py
@@ -308,12 +308,12 @@ class DefaultOSUtil(object):
         '''
         if os.path.isfile(PRODUCT_ID_FILE):
             s = fileutil.read_file(PRODUCT_ID_FILE).strip()
-
+            
         else:
             rc, s = shellutil.run_get_output(DMIDECODE_CMD)
             if rc != 0 or UUID_PATTERN.match(s) is None:
                 return ""
-
+              
         return self._correct_instance_id(s.strip())
 
     def get_userentry(self, username):
@@ -390,12 +390,12 @@ class DefaultOSUtil(object):
         if not remove:
             # for older distros create sudoers.d
             if not os.path.isdir(sudoers_dir):
-                sudoers_file = os.path.join(sudoers_dir, '../sudoers')
                 # create the sudoers.d directory
-                os.mkdir(sudoers_dir)
+                fileutil.mkdir(sudoers_dir)
                 # add the include of sudoers.d to the /etc/sudoers
-                sudoers = '\n#includedir ' + sudoers_dir + '\n'
-                fileutil.append_file(sudoers_file, sudoers)
+                sudoers_file = os.path.join(sudoers_dir, os.pardir, 'sudoers')
+                include_sudoers_dir = "\n#includedir {0}\n".format(sudoers_dir)
+                fileutil.append_file(sudoers_file, include_sudoers_dir)
             sudoer = None
             if nopasswd:
                 sudoer = "{0} ALL=(ALL) NOPASSWD: ALL".format(username)
@@ -1012,6 +1012,9 @@ class DefaultOSUtil(object):
         shellutil.run("route del 255.255.255.255 dev {0}".format(ifname),
                       chk_err=False)
 
+    def is_dhcp_available(self):
+        return (True, '')
+
     def is_dhcp_enabled(self):
         return False
 
diff --git a/azurelinuxagent/common/osutil/factory.py b/azurelinuxagent/common/osutil/factory.py
index 223c3a2b..480f2a81 100644
--- a/azurelinuxagent/common/osutil/factory.py
+++ b/azurelinuxagent/common/osutil/factory.py
@@ -105,6 +105,9 @@ def get_osutil(distro_name=DISTRO_NAME,
     elif distro_name == "gaia":
         return GaiaOSUtil()
 
+    if distro_name == "iosxe":
+        return IosxeOSUtil()
+
     else:
         logger.warn("Unable to load distro implementation for {0}. Using "
                     "default distro implementation instead.",
diff --git a/azurelinuxagent/common/osutil/iosxe.py b/azurelinuxagent/common/osutil/iosxe.py
new file mode 100644
index 00000000..3152ada8
--- /dev/null
+++ b/azurelinuxagent/common/osutil/iosxe.py
@@ -0,0 +1,84 @@
+# Microsoft Azure Linux Agent
+#
+# Copyright 2018 Microsoft Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Requires Python 2.6+ and Openssl 1.0+
+#
+
+import azurelinuxagent.common.logger as logger
+import azurelinuxagent.common.utils.shellutil as shellutil
+from azurelinuxagent.common.osutil.default import DefaultOSUtil
+from azurelinuxagent.common.osutil.redhat import Redhat6xOSUtil
+
+'''
+The IOSXE distribution is a variant of the Centos distribution, 
+version 7.1.
+The primary difference is that IOSXE makes some assumptions about
+the waagent environment:
+ - only the waagent daemon is executed
+ - no provisioning is performed
+ - no DHCP-based services are available
+'''
+class IosxeOSUtil(DefaultOSUtil):
+    def __init__(self):
+        super(IosxeOSUtil, self).__init__()
+
+    def set_hostname(self, hostname):
+        """
+        Unlike redhat 6.x, redhat 7.x will set hostname via hostnamectl
+        Due to a bug in systemd in Centos-7.0, if this call fails, fallback
+        to hostname.
+        """
+        hostnamectl_cmd = "hostnamectl set-hostname {0} --static".format(hostname)
+        if shellutil.run(hostnamectl_cmd, chk_err=False) != 0:
+            logger.warn("[{0}] failed, attempting fallback".format(hostnamectl_cmd))
+            DefaultOSUtil.set_hostname(self, hostname)
+
+    def publish_hostname(self, hostname):
+        """
+        Restart NetworkManager first before publishing hostname
+        """
+        shellutil.run("service NetworkManager restart")
+        super(RedhatOSUtil, self).publish_hostname(hostname)
+
+    def register_agent_service(self):
+        return shellutil.run("systemctl enable waagent", chk_err=False)
+
+    def unregister_agent_service(self):
+        return shellutil.run("systemctl disable waagent", chk_err=False)
+
+    def openssl_to_openssh(self, input_file, output_file):
+        DefaultOSUtil.openssl_to_openssh(self, input_file, output_file)
+
+    def is_dhcp_available(self):
+        return (False, '168.63.129.16')
+
+    def get_instance_id(self):
+        '''
+        Azure records a UUID as the instance ID
+        First check /sys/class/dmi/id/product_uuid.
+        If that is missing, then extracts from dmidecode
+        If nothing works (for old VMs), return the empty string
+        '''
+        if os.path.isfile(PRODUCT_ID_FILE):
+            try:
+                s = fileutil.read_file(PRODUCT_ID_FILE).strip()
+                return self._correct_instance_id(s.strip())
+            except IOError:
+                pass
+        rc, s = shellutil.run_get_output(DMIDECODE_CMD)
+        if rc != 0 or UUID_PATTERN.match(s) is None:
+            return ""
+        return self._correct_instance_id(s.strip())
diff --git a/azurelinuxagent/common/protocol/restapi.py b/azurelinuxagent/common/protocol/restapi.py
index 771006d1..00da1cde 100644
--- a/azurelinuxagent/common/protocol/restapi.py
+++ b/azurelinuxagent/common/protocol/restapi.py
@@ -161,8 +161,6 @@ class Extension(DataContract):
 class ExtHandlerProperties(DataContract):
     def __init__(self):
         self.version = None
-        self.upgradePolicy = None
-        self.upgradeGuid = None
         self.dependencyLevel = None
         self.state = None
         self.extensions = DataContractList(Extension)
@@ -257,13 +255,11 @@ class ExtHandlerStatus(DataContract):
     def __init__(self,
                  name=None,
                  version=None,
-                 upgradeGuid=None,
                  status=None,
                  code=0,
                  message=None):
         self.name = name
         self.version = version
-        self.upgradeGuid = upgradeGuid
         self.status = status
         self.code = code
         self.message = message
diff --git a/azurelinuxagent/common/protocol/util.py b/azurelinuxagent/common/protocol/util.py
index a3e3176f..b642b13e 100644
--- a/azurelinuxagent/common/protocol/util.py
+++ b/azurelinuxagent/common/protocol/util.py
@@ -146,13 +146,26 @@ class ProtocolUtil(object):
     def _detect_wire_protocol(self):
         endpoint = self.dhcp_handler.endpoint
         if endpoint is None:
-            logger.info("WireServer endpoint is not found. Rerun dhcp handler")
-            try:
-                self.dhcp_handler.run()
-            except DhcpError as e:
-                raise ProtocolError(ustr(e))
-            endpoint = self.dhcp_handler.endpoint
-        
+            '''
+            Check if DHCP can be used to get the wire protocol endpoint
+            '''
+            (dhcp_available, conf_endpoint) =  self.osutil.is_dhcp_available()
+            if dhcp_available:
+                logger.info("WireServer endpoint is not found. Rerun dhcp handler")
+                try:
+                    self.dhcp_handler.run()
+                except DhcpError as e:
+                    raise ProtocolError(ustr(e))
+                endpoint = self.dhcp_handler.endpoint
+            else:
+                logger.info("_detect_wire_protocol: DHCP not available")
+                endpoint = self._get_wireserver_endpoint()
+                if endpoint == None:
+                    endpoint = conf_endpoint
+                    logger.info("Using hardcoded WireServer endpoint {0}", endpoint)
+                else:
+                    logger.info("WireServer endpoint {0} read from file", endpoint)
+
         try:
             protocol = WireProtocol(endpoint)
             protocol.detect()
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index bf423ce8..14fbadb1 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -328,7 +328,8 @@ def ext_handler_status_to_v1(handler_status, ext_statuses, timestamp):
         'handlerVersion': handler_status.version,
         'handlerName': handler_status.name,
         'status': handler_status.status,
-        'code': handler_status.code
+        'code': handler_status.code,
+        'useExactVersion': True
     }
     if handler_status.message is not None:
         v1_handler_status["formattedMessage"] = {
@@ -336,9 +337,6 @@ def ext_handler_status_to_v1(handler_status, ext_statuses, timestamp):
             "message": handler_status.message
         }
 
-    if handler_status.upgradeGuid is not None:
-        v1_handler_status["upgradeGuid"] = handler_status.upgradeGuid
-
     if len(handler_status.extensions) > 0:
         # Currently, no more than one extension per handler
         ext_name = handler_status.extensions[0]
@@ -1568,21 +1566,11 @@ class ExtensionsConfig(object):
         ext_handler.properties.version = getattrib(plugin, "version")
         ext_handler.properties.state = getattrib(plugin, "state")
 
-        ext_handler.properties.upgradeGuid = getattrib(plugin, "upgradeGuid")
-        if not ext_handler.properties.upgradeGuid:
-            ext_handler.properties.upgradeGuid = None
-
         try:
             ext_handler.properties.dependencyLevel = int(getattrib(plugin, "dependencyLevel"))
         except ValueError:
             ext_handler.properties.dependencyLevel = 0
 
-        auto_upgrade = getattrib(plugin, "autoUpgrade")
-        if auto_upgrade is not None and auto_upgrade.lower() == "true":
-            ext_handler.properties.upgradePolicy = "auto"
-        else:
-            ext_handler.properties.upgradePolicy = "manual"
-
         location = getattrib(plugin, "location")
         failover_location = getattrib(plugin, "failoverlocation")
         for uri in [location, failover_location]:
diff --git a/azurelinuxagent/common/rdma.py b/azurelinuxagent/common/rdma.py
index 3c01e776..5504a068 100644
--- a/azurelinuxagent/common/rdma.py
+++ b/azurelinuxagent/common/rdma.py
@@ -84,6 +84,8 @@ class RDMAHandler(object):
         """Retrieve the firmware version information from the system.
            This depends on information provided by the Linux kernel."""
 
+        kvp_key_size = 512
+        kvp_value_size = 2048
         driver_info_source = '/var/lib/hyperv/.kvp_pool_0'
         base_kernel_err_msg = 'Kernel does not provide the necessary '
         base_kernel_err_msg += 'information or the kvp daemon is not running.'
@@ -93,21 +95,23 @@ class RDMAHandler(object):
             logger.error(error_msg % driver_info_source)
             return
 
-        lines = open(driver_info_source).read()
-        if not lines:
-            error_msg = 'RDMA: Source file "%s" is empty. '
-            error_msg += base_kernel_err_msg
-            logger.error(error_msg % driver_info_source)
-            return
-
-        r = re.search("NdDriverVersion\0+(\d\d\d\.\d)", lines)
-        if r:
-            NdDriverVersion = r.groups()[0]
-            return NdDriverVersion
-        else:
-            error_msg = 'RDMA: NdDriverVersion not found in "%s"'
-            logger.error(error_msg % driver_info_source)
-            return
+        f = open(driver_info_source)
+        while True :
+            key = f.read(kvp_key_size)
+            value = f.read(kvp_value_size)
+            if key and value :
+                key_0 = key.split("\x00")[0]
+                value_0 = value.split("\x00")[0]
+                if key_0 == "NdDriverVersion" :
+                    f.close()
+                    return value_0
+            else :
+                break
+        f.close()
+
+        error_msg = 'RDMA: NdDriverVersion not found in "%s"'
+        logger.error(error_msg % driver_info_source)
+        return
 
     @staticmethod
     def is_kvp_daemon_running():
diff --git a/azurelinuxagent/common/utils/flexible_version.py b/azurelinuxagent/common/utils/flexible_version.py
index 14c2a733..81202ac9 100644
--- a/azurelinuxagent/common/utils/flexible_version.py
+++ b/azurelinuxagent/common/utils/flexible_version.py
@@ -147,6 +147,18 @@ class FlexibleVersion(version.Version):
 
         return True
 
+    def matches(self, that):
+        if self.sep != that.sep or len(self.version) > len(that.version):
+            return False
+
+        for i in range(len(self.version)):
+            if self.version[i] != that.version[i]:
+                return False
+        if self.prerel_tags:
+            return self.prerel_tags == that.prerel_tags
+
+        return True
+
     def _assemble(self, version, sep, prerel_sep, prerelease):
         s = sep.join(map(str, version))
         if prerelease is not None:
diff --git a/azurelinuxagent/common/utils/restutil.py b/azurelinuxagent/common/utils/restutil.py
index 13fc08b3..dee51e0e 100644
--- a/azurelinuxagent/common/utils/restutil.py
+++ b/azurelinuxagent/common/utils/restutil.py
@@ -18,6 +18,7 @@
 #
 
 import os
+import re
 import threading
 import time
 import traceback
@@ -38,6 +39,9 @@ DELAY_IN_SECONDS = 1
 THROTTLE_RETRIES = 25
 THROTTLE_DELAY_IN_SECONDS = 1
 
+REDACTED_TEXT = "<SAS_SIGNATURE>"
+SAS_TOKEN_RETRIEVAL_REGEX = re.compile(r'^(https?://[a-zA-Z0-9.].*sig=)([a-zA-Z0-9%-]*)(.*)$')
+
 RETRY_CODES = [
     httpclient.RESET_CONTENT,
     httpclient.PARTIAL_CONTENT,
@@ -170,6 +174,10 @@ def _get_http_proxy(secure=False):
     return host, port
 
 
+def redact_sas_tokens_in_urls(url):
+    return SAS_TOKEN_RETRIEVAL_REGEX.sub(r"\1" + REDACTED_TEXT + r"\3", url)
+
+
 def _http_request(method, host, rel_uri, port=None, data=None, secure=False,
                   headers=None, proxy_host=None, proxy_port=None):
 
@@ -205,7 +213,7 @@ def _http_request(method, host, rel_uri, port=None, data=None, secure=False,
 
     logger.verbose("HTTP connection [{0}] [{1}] [{2}] [{3}]",
                    method,
-                   url,
+                   redact_sas_tokens_in_urls(url),
                    data,
                    headers)
 
@@ -316,7 +324,7 @@ def http_request(method,
 
         except httpclient.HTTPException as e:
             msg = '[HTTP Failed] {0} {1} -- HttpException {2}'.format(
-                method, url, e)
+                method, redact_sas_tokens_in_urls(url), e)
             if _is_retry_exception(e):
                 continue
             break
@@ -324,7 +332,7 @@ def http_request(method,
         except IOError as e:
             IOErrorCounter.increment(host=host, port=port)
             msg = '[HTTP Failed] {0} {1} -- IOError {2}'.format(
-                method, url, e)
+                method, redact_sas_tokens_in_urls(url), e)
             continue
 
     raise HttpError("{0} -- {1} attempts made".format(msg,attempt))
diff --git a/azurelinuxagent/common/version.py b/azurelinuxagent/common/version.py
index 4d10b024..ecbd7c0f 100644
--- a/azurelinuxagent/common/version.py
+++ b/azurelinuxagent/common/version.py
@@ -85,9 +85,9 @@ def get_distro():
     else:
         try:
             # dist() removed in Python 3.7
-            osinfo = platform.dist()
+            osinfo = list(platform.dist()) + ['']
         except:
-            osinfo = ('UNKNOWN', 'FFFF', '')
+            osinfo = ['UNKNOWN', 'FFFF', '', '']
 
     # The platform.py lib has issue with detecting oracle linux distribution.
     # Merge the following patch provided by oracle as a temporary fix.
@@ -106,6 +106,9 @@ def get_distro():
     if os.path.exists("/etc/cp-release"):
         osinfo = get_checkpoint_platform()
 
+    if os.path.exists("/home/guestshell/azure"):
+        osinfo = ['iosxe', 'csr1000v', '', 'Cisco IOSXE Linux']
+
     # Remove trailing whitespace and quote in distro name
     osinfo[0] = osinfo[0].strip('"').strip(' ').lower()
     return osinfo
diff --git a/azurelinuxagent/ga/exthandlers.py b/azurelinuxagent/ga/exthandlers.py
index f46447ab..a66aacbd 100644
--- a/azurelinuxagent/ga/exthandlers.py
+++ b/azurelinuxagent/ga/exthandlers.py
@@ -185,7 +185,6 @@ class ExtHandlersHandler(object):
         self.protocol = None
         self.ext_handlers = None
         self.last_etag = None
-        self.last_upgrade_guids = {}
         self.log_report = False
         self.log_etag = True
         self.log_process = False
@@ -236,23 +235,6 @@ class ExtHandlersHandler(object):
                       message=msg)
             return
 
-    def get_upgrade_guid(self, name):
-        return self.last_upgrade_guids.get(name, (None, False))[0]
-
-    def get_log_upgrade_guid(self, ext_handler):
-        return self.last_upgrade_guids.get(ext_handler.name, (None, False))[1]
-
-    def set_log_upgrade_guid(self, ext_handler, log_val):
-        guid = self.get_upgrade_guid(ext_handler.name)
-        if guid is not None:
-            self.last_upgrade_guids[ext_handler.name] = (guid, log_val)
-
-    def is_new_guid(self, ext_handler):
-        last_guid = self.get_upgrade_guid(ext_handler.name)
-        if last_guid is None:
-            return True
-        return last_guid != ext_handler.properties.upgradeGuid
-
     def cleanup_outdated_handlers(self):
         handlers = []
         pkgs = []
@@ -333,28 +315,22 @@ class ExtHandlersHandler(object):
 
         try:
             state = ext_handler.properties.state
-            # The extension is to be enabled, there is an upgrade GUID
-            # and the GUID is NOT new
-            if state == u"enabled" and \
-                    ext_handler.properties.upgradeGuid is not None and \
-                    not self.is_new_guid(ext_handler):
-                ext_handler_i.ext_handler.properties.version = ext_handler_i.get_installed_version()
-                ext_handler_i.set_logger()
-                if self.last_etag != etag:
-                    self.set_log_upgrade_guid(ext_handler, True)
-
-                msg = "New GUID is the same as the old GUID. Exiting without upgrading."
-                if self.get_log_upgrade_guid(ext_handler):
-                    ext_handler_i.logger.info(msg)
-                    self.set_log_upgrade_guid(ext_handler, False)
-                ext_handler_i.set_handler_state(ExtHandlerState.Enabled)
-                ext_handler_i.set_handler_status(status="Ready", message="No change")
-                ext_handler_i.set_operation(WALAEventOperation.SkipUpdate)
-                ext_handler_i.report_event(message=ustr(msg), is_success=True)
-                return
-
-            self.set_log_upgrade_guid(ext_handler, True)
-            ext_handler_i.decide_version(target_state=state)
+            max_retries = 5
+            retry_count = 0
+            while ext_handler_i.decide_version(target_state=state) is None:
+                if retry_count >= max_retries:
+                    err_msg = "Unable to find manifest for extension {0}".format(ext_handler_i.ext_handler.name)
+                    ext_handler_i.set_operation(WALAEventOperation.Download)
+                    ext_handler_i.set_handler_status(message=ustr(err_msg), code=-1)
+                    ext_handler_i.report_event(message=ustr(err_msg), is_success=False)
+                    return
+                time.sleep(2**retry_count * 10)
+                retry_count += 1
+            if retry_count != 0:
+                ext_handler_i.set_operation(WALAEventOperation.Download)
+                err_msg = "Able to find manifest for extension {0} after {1} failed attempts.".format(
+                    ext_handler_i.ext_handler.name, retry_count)
+                ext_handler_i.report_event(message=ustr(err_msg))
             self.get_artifact_error_state.reset()
             if not ext_handler_i.is_upgrade and self.last_etag == etag:
                 if self.log_etag:
@@ -369,44 +345,35 @@ class ExtHandlersHandler(object):
             ext_handler_i.logger.info("Target handler state: {0}", state)
             if state == u"enabled":
                 self.handle_enable(ext_handler_i)
-                if ext_handler.properties.upgradeGuid is not None:
-                    ext_handler_i.logger.info("New Upgrade GUID: {0}", ext_handler.properties.upgradeGuid)
-                    self.last_upgrade_guids[ext_handler.name] = (ext_handler.properties.upgradeGuid, True)
             elif state == u"disabled":
                 self.handle_disable(ext_handler_i)
-                # Remove the GUID from the dictionary so that it is upgraded upon re-enable
-                self.last_upgrade_guids.pop(ext_handler.name, None)
             elif state == u"uninstall":
                 self.handle_uninstall(ext_handler_i)
-                # Remove the GUID from the dictionary so that it is upgraded upon re-install
-                self.last_upgrade_guids.pop(ext_handler.name, None)
             else:
                 message = u"Unknown ext handler state:{0}".format(state)
                 raise ExtensionError(message)
+        except ExtensionError as e:
+            self.handle_handle_ext_handler_error(ext_handler_i, e, e.code)
         except Exception as e:
-            msg = ustr(e)
-            ext_handler_i.set_handler_status(message=msg, code=-1)
-
-            self.get_artifact_error_state.incr()
-            if self.get_artifact_error_state.is_triggered():
-                report_event(op=WALAEventOperation.GetArtifactExtended,
-                             message="Failed to get artifact for over "
-                                     "{0}: {1}".format(self.get_artifact_error_state.min_timedelta, msg),
-                             is_success=False)
-                self.get_artifact_error_state.reset()
-            else:
-                ext_handler_i.logger.warn(msg)
+            self.handle_handle_ext_handler_error(ext_handler_i, e)
+
+    def handle_handle_ext_handler_error(self, ext_handler_i, e, code=-1):
+        msg = ustr(e)
+        ext_handler_i.set_handler_status(message=msg, code=code)
+
+        self.get_artifact_error_state.incr()
+        if self.get_artifact_error_state.is_triggered():
+            report_event(op=WALAEventOperation.GetArtifactExtended,
+                         message="Failed to get artifact for over "
+                                 "{0}: {1}".format(self.get_artifact_error_state.min_timedelta, msg),
+                         is_success=False)
+            self.get_artifact_error_state.reset()
+        else:
+            ext_handler_i.logger.warn(msg)
     
     def handle_enable(self, ext_handler_i):
         self.log_process = True
         old_ext_handler_i = ext_handler_i.get_installed_ext_handler()
-        if old_ext_handler_i is not None and \
-           old_ext_handler_i.version_gt(ext_handler_i):
-            msg = "Downgrade is not allowed. Skipping install and enable."
-            ext_handler_i.logger.error(msg)
-            ext_handler_i.set_operation(WALAEventOperation.Downgrade)
-            ext_handler_i.report_event(message=ustr(msg), is_success=True)
-            return
 
         handler_state = ext_handler_i.get_handler_state()
         ext_handler_i.logger.info("[Enable] current handler state is: {0}",
@@ -417,10 +384,13 @@ class ExtHandlersHandler(object):
             ext_handler_i.update_settings()
             if old_ext_handler_i is None:
                 ext_handler_i.install()
-            elif ext_handler_i.version_gt(old_ext_handler_i):
+            elif ext_handler_i.version_ne(old_ext_handler_i):
                 old_ext_handler_i.disable()
                 ext_handler_i.copy_status_files(old_ext_handler_i)
-                ext_handler_i.update()
+                if ext_handler_i.version_gt(old_ext_handler_i):
+                    ext_handler_i.update()
+                else:
+                    old_ext_handler_i.update(version=ext_handler_i.ext_handler.properties.version)
                 old_ext_handler_i.uninstall()
                 old_ext_handler_i.rm_ext_handler_dir()
                 ext_handler_i.update_with_install()
@@ -502,9 +472,6 @@ class ExtHandlersHandler(object):
         handler_status = ext_handler_i.get_handler_status() 
         if handler_status is None:
             return
-        guid = self.get_upgrade_guid(ext_handler.name)
-        if guid is not None:
-            handler_status.upgradeGuid = guid
 
         handler_state = ext_handler_i.get_handler_state()
         if handler_state != ExtHandlerState.NotInstalled:
@@ -562,74 +529,21 @@ class ExtHandlerInstance(object):
         # - Find the installed package (its version must exactly match)
         # - Find the internal candidate (its version must exactly match)
         # - Separate the public packages
-        internal_pkg = None
+        selected_pkg = None
         installed_pkg = None
-        public_pkgs = []
+        pkg_list.versions.sort(key=lambda p: FlexibleVersion(p.version))
         for pkg in pkg_list.versions:
             pkg_version = FlexibleVersion(pkg.version)
             if pkg_version == installed_version:
                 installed_pkg = pkg
-            if pkg.isinternal and pkg_version == requested_version:
-                internal_pkg = pkg
-            if not pkg.isinternal:
-                public_pkgs.append(pkg)
-
-        internal_version = FlexibleVersion(internal_pkg.version) \
-                            if internal_pkg is not None \
-                            else FlexibleVersion()
-        public_pkgs.sort(key=lambda p: FlexibleVersion(p.version), reverse=True)
-
-        # Determine the preferred version and type of upgrade occurring
-        preferred_version = max(requested_version, installed_version)
-        is_major_upgrade = preferred_version.major > installed_version.major
-        allow_minor_upgrade = self.ext_handler.properties.upgradePolicy == 'auto'
-
-        # Find the first public candidate which
-        # - Matches the preferred major version
-        # - Does not upgrade to a new, disallowed major version
-        # - And only increments the minor version if allowed
-        # Notes:
-        # - The patch / hotfix version is not considered
-        public_pkg = None
-        for pkg in public_pkgs:
-            pkg_version = FlexibleVersion(pkg.version)
-            if pkg_version.major == preferred_version.major \
-                    and (not pkg.disallow_major_upgrade or not is_major_upgrade) \
-                    and (allow_minor_upgrade or pkg_version.minor == preferred_version.minor):
-                public_pkg = pkg
-                break
-
-        # If there are no candidates, locate the highest public version whose
-        # major matches that installed
-        if internal_pkg is None and public_pkg is None:
-            for pkg in public_pkgs:
-                pkg_version = FlexibleVersion(pkg.version)
-                if pkg_version.major == installed_version.major:
-                    public_pkg = pkg
-                    break
-
-        public_version = FlexibleVersion(public_pkg.version) \
-                            if public_pkg is not None \
-                            else FlexibleVersion()
-
-        # Select the candidate
-        # - Use the public candidate if there is no internal candidate or
-        #   the public is more recent (e.g., a hotfix patch)
-        # - Otherwise use the internal candidate
-        if internal_pkg is None or (public_pkg is not None and public_version > internal_version):
-            selected_pkg = public_pkg
-        else:
-            selected_pkg = internal_pkg
-
-        selected_version = FlexibleVersion(selected_pkg.version) \
-                            if selected_pkg is not None \
-                            else FlexibleVersion()
+            if requested_version.matches(pkg_version):
+                selected_pkg = pkg
 
         # Finally, update the version only if not downgrading
         # Note:
         #  - A downgrade, which will be bound to the same major version,
         #    is allowed if the installed version is no longer available
-        if target_state == u"uninstall":
+        if target_state == u"uninstall" or target_state == u"disabled":
             if installed_pkg is None:
                 msg = "Failed to find installed version of {0} " \
                     "to uninstall".format(self.ext_handler.name)
@@ -637,26 +551,20 @@ class ExtHandlerInstance(object):
             self.pkg = installed_pkg
             self.ext_handler.properties.version = str(installed_version) \
                                     if installed_version is not None else None
-        elif selected_pkg is None \
-                or (installed_pkg is not None and selected_version < installed_version):
-            self.pkg = installed_pkg
-            self.ext_handler.properties.version = str(installed_version) \
-                                    if installed_version is not None else None
         else:
             self.pkg = selected_pkg
-            self.ext_handler.properties.version = str(selected_pkg.version)
+            if self.pkg is not None:
+                self.ext_handler.properties.version = str(selected_pkg.version)
 
-        # Note if the selected package is greater than that installed
+        # Note if the selected package is different than that installed
         if installed_pkg is None \
-                or FlexibleVersion(self.pkg.version) > FlexibleVersion(installed_pkg.version):
+                or (self.pkg is not None and FlexibleVersion(self.pkg.version) != FlexibleVersion(installed_pkg.version)):
             self.is_upgrade = True
 
-        if self.pkg is None:
-            raise ExtensionError("Failed to find any valid extension package")
-
-        self.logger.verbose("Use version: {0}", self.pkg.version)
+        if self.pkg is not None:
+            self.logger.verbose("Use version: {0}", self.pkg.version)
         self.set_logger()
-        return
+        return self.pkg
 
     def set_logger(self):
         prefix = "[{0}]".format(self.get_full_name())
@@ -667,6 +575,11 @@ class ExtHandlerInstance(object):
         other_version = other.ext_handler.properties.version
         return FlexibleVersion(self_version) > FlexibleVersion(other_version)
 
+    def version_ne(self, other):
+        self_version = self.ext_handler.properties.version
+        other_version = other.ext_handler.properties.version
+        return FlexibleVersion(self_version) != FlexibleVersion(other_version)
+
     def get_installed_ext_handler(self):
         lastest_version = self.get_installed_version()
         if lastest_version is None:
@@ -819,7 +732,7 @@ class ExtHandlerInstance(object):
         man = self.load_manifest()
         enable_cmd = man.get_enable_command()
         self.logger.info("Enable extension [{0}]".format(enable_cmd))
-        self.launch_command(enable_cmd, timeout=300)
+        self.launch_command(enable_cmd, timeout=300, extension_error_code=1009)
         self.set_handler_state(ExtHandlerState.Enabled)
         self.set_handler_status(status="Ready", message="Plugin enabled")
 
@@ -828,7 +741,8 @@ class ExtHandlerInstance(object):
         man = self.load_manifest()
         disable_cmd = man.get_disable_command()
         self.logger.info("Disable extension [{0}]".format(disable_cmd))
-        self.launch_command(disable_cmd, timeout=900)
+        self.launch_command(disable_cmd, timeout=900,
+                            extension_error_code=1010)
         self.set_handler_state(ExtHandlerState.Installed)
         self.set_handler_status(status="NotReady", message="Plugin disabled")
 
@@ -837,7 +751,7 @@ class ExtHandlerInstance(object):
         install_cmd = man.get_install_command()
         self.logger.info("Install extension [{0}]".format(install_cmd))
         self.set_operation(WALAEventOperation.Install)
-        self.launch_command(install_cmd, timeout=900)
+        self.launch_command(install_cmd, timeout=900, extension_error_code=1007)
         self.set_handler_state(ExtHandlerState.Installed)
 
     def uninstall(self):
@@ -862,12 +776,18 @@ class ExtHandlerInstance(object):
             self.report_event(message=message, is_success=False)
             self.logger.warn(message)
 
-    def update(self):
+    def update(self, version=None):
+        if version is None:
+            version = self.ext_handler.properties.version
+
         self.set_operation(WALAEventOperation.Update)
         man = self.load_manifest()
         update_cmd = man.get_update_command()
         self.logger.info("Update extension [{0}]".format(update_cmd))
-        self.launch_command(update_cmd, timeout=900)
+        self.launch_command(update_cmd, 
+                            timeout=900,
+                            extension_error_code=1008, 
+                            env={'VERSION': version})
     
     def update_with_install(self):
         man = self.load_manifest()
@@ -980,11 +900,15 @@ class ExtHandlerInstance(object):
         last_update = int(time.time() - os.stat(heartbeat_file).st_mtime)
         return last_update <= 600
 
-    def launch_command(self, cmd, timeout=300):
+    def launch_command(self, cmd, timeout=300, extension_error_code=-1, env=None):
         begin_utc = datetime.datetime.utcnow()
         self.logger.verbose("Launch command: [{0}]", cmd)
         base_dir = self.get_base_dir()
 
+        if env is None:
+            env = {}
+        env.update(os.environ)
+
         try:
             # This should be .run(), but due to the wide variety
             # of Python versions we must support we must use .communicate().
@@ -1006,10 +930,11 @@ class ExtHandlerInstance(object):
                                        cwd=base_dir,
                                        stdout=subprocess.PIPE,
                                        stderr=subprocess.PIPE,
-                                       env=os.environ,
+                                       env=env,
                                        preexec_fn=pre_exec_function)
         except OSError as e:
-            raise ExtensionError("Failed to launch '{0}': {1}".format(full_path, e.strerror))
+            raise ExtensionError("Failed to launch '{0}': {1}".format(full_path, e.strerror),
+                                 code=extension_error_code)
 
         cg = CGroups.for_extension(self.ext_handler.name)
         CGroupsTelemetry.track_extension(self.ext_handler.name, cg)
@@ -1017,9 +942,11 @@ class ExtHandlerInstance(object):
 
         ret = process.poll()
         if ret is None:
-            raise ExtensionError("Process {0} was not terminated: {1}\n{2}".format(process.pid, cmd, msg))
+            raise ExtensionError("Process {0} was not terminated: {1}\n{2}".format(process.pid, cmd, msg),
+                                 code=extension_error_code)
         if ret != 0:
-            raise ExtensionError("Non-zero exit code: {0}, {1}\n{2}".format(ret, cmd, msg))
+            raise ExtensionError("Non-zero exit code: {0}, {1}\n{2}".format(ret, cmd, msg),
+                                 code=extension_error_code)
 
         duration = elapsed_milliseconds(begin_utc)
         self.report_event(message="{0}\n{1}".format(cmd, msg), duration=duration, log_event=False)
diff --git a/azurelinuxagent/ga/remoteaccess.py b/azurelinuxagent/ga/remoteaccess.py
index 9be2176b..d11d39e3 100644
--- a/azurelinuxagent/ga/remoteaccess.py
+++ b/azurelinuxagent/ga/remoteaccess.py
@@ -46,7 +46,7 @@ from pwd import getpwall
 from azurelinuxagent.common.errorstate import ErrorState
 
 from azurelinuxagent.common.event import add_event, WALAEventOperation, elapsed_milliseconds
-from azurelinuxagent.common.exception import ExtensionError, ProtocolError
+from azurelinuxagent.common.exception import ExtensionError, ProtocolError, RemoteAccessError
 from azurelinuxagent.common.future import ustr
 from azurelinuxagent.common.protocol.restapi import ExtHandlerStatus, \
                                                     ExtensionStatus, \
@@ -80,6 +80,7 @@ class RemoteAccessHandler(object):
         self.cryptUtil = CryptUtil(conf.get_openssl_cmd())
         self.remote_access = None
         self.incarnation = 0
+        self.error_message = ""
 
     def run(self):
         try:
@@ -90,8 +91,7 @@ class RemoteAccessHandler(object):
                     # something changed. Handle remote access if any.
                     self.incarnation = current_incarnation
                     self.remote_access = self.protocol.client.get_remote_access()
-                    if self.remote_access is not None:
-                        self.handle_remote_access()
+                    self.handle_remote_access()
         except Exception as e:
             msg = u"Exception processing remote access handler: {0} {1}".format(ustr(e), traceback.format_exc())
             logger.error(msg)
@@ -102,19 +102,44 @@ class RemoteAccessHandler(object):
                       message=msg)
 
     def handle_remote_access(self):
+        # Get JIT user accounts.
+        all_users = self.os_util.get_users()
+        existing_jit_users = set(u[0] for u in all_users if self.validate_jit_user(u[4]))
+        self.err_message = ""
         if self.remote_access is not None:
-            # Get JIT user accounts.
-            all_users = self.os_util.get_users()
-            jit_users = set()
-            for usr in all_users:
-                if self.validate_jit_user(usr[4]):
-                    jit_users.add(usr[0])
+            goal_state_users = set(u.name for u in self.remote_access.user_list.users)
             for acc in self.remote_access.user_list.users:
-                raw_expiration = acc.expiration
-                account_expiration = datetime.strptime(raw_expiration, REMOTE_USR_EXPIRATION_FORMAT)
-                now = datetime.utcnow()
-                if acc.name not in jit_users and now < account_expiration:
-                    self.add_user(acc.name, acc.encrypted_password, account_expiration)
+                try:
+                    raw_expiration = acc.expiration
+                    account_expiration = datetime.strptime(raw_expiration, REMOTE_USR_EXPIRATION_FORMAT)
+                    now = datetime.utcnow()
+                    if acc.name not in existing_jit_users and now < account_expiration:
+                        self.add_user(acc.name, acc.encrypted_password, account_expiration)
+                    elif acc.name in existing_jit_users and now > account_expiration:
+                        # user account expired, delete it.
+                        logger.info("user {0} expired from remote_access".format(acc.name))
+                        self.remove_user(acc.name)
+                except RemoteAccessError as rae:
+                    self.err_message = self.err_message + "Error processing user {0}. Exception: {1}"\
+                        .format(acc.name, ustr(rae))
+            for user in existing_jit_users:
+                try:
+                    if user not in goal_state_users:
+                        # user explicitly removed
+                        logger.info("User {0} removed from remote_access".format(user))
+                        self.remove_user(user)
+                except RemoteAccessError as rae:
+                    self.err_message = self.err_message + "Error removing user {0}. Exception: {1}"\
+                        .format(user, ustr(rae))
+        else:
+            # All users removed, remove any remaining JIT accounts.
+            for user in existing_jit_users:
+                try:
+                    logger.info("User {0} removed from remote_access. remote_access empty".format(user))
+                    self.remove_user(user)
+                except RemoteAccessError as rae:
+                    self.err_message = self.err_message + "Error removing user {0}. Exception: {1}"\
+                        .format(user, ustr(rae))
 
     def validate_jit_user(self, comment):
         return comment == REMOTE_ACCESS_ACCOUNT_COMMENT
@@ -122,40 +147,36 @@ class RemoteAccessHandler(object):
     def add_user(self, username, encrypted_password, account_expiration):
         try:
             expiration_date = (account_expiration + timedelta(days=1)).strftime(DATE_FORMAT)
-            logger.verbose("Adding user {0} with expiration date {1}"
-                           .format(username, expiration_date))
+            logger.verbose("Adding user {0} with expiration date {1}".format(username, expiration_date))
             self.os_util.useradd(username, expiration_date, REMOTE_ACCESS_ACCOUNT_COMMENT)
-        except OSError as oe:
-            logger.error("Error adding user {0}. {1}"
-                         .format(username, oe.strerror))
-            return
         except Exception as e:
-            logger.error("Error adding user {0}. {1}".format(username, ustr(e)))
-            return
+            raise RemoteAccessError("Error adding user {0}. {1}".format(username, ustr(e)))
         try:
             prv_key = os.path.join(conf.get_lib_dir(), TRANSPORT_PRIVATE_CERT)
             pwd = self.cryptUtil.decrypt_secret(encrypted_password, prv_key)
             self.os_util.chpasswd(username, pwd, conf.get_password_cryptid(), conf.get_password_crypt_salt_len())
             self.os_util.conf_sudoer(username)
-            logger.info("User '{0}' added successfully with expiration in {1}"
-                        .format(username, expiration_date))
-            return
-        except OSError as oe:
-            self.handle_failed_create(username, oe.strerror)
+            logger.info("User '{0}' added successfully with expiration in {1}".format(username, expiration_date))
         except Exception as e:
-            self.handle_failed_create(username, ustr(e))
+            error = "Error adding user {0}. {1} ".format(username, str(e))
+            try:
+                self.handle_failed_create(username)
+                error += "cleanup successful"
+            except RemoteAccessError as rae:
+                error += "and error cleaning up {0}".format(str(rae))
+            raise RemoteAccessError("Error adding user {0} cleanup successful".format(username), ustr(e))
 
-    def handle_failed_create(self, username, error_message):
-        logger.error("Error creating user {0}. {1}"
-                     .format(username, error_message))
+    def handle_failed_create(self, username):
         try:
             self.delete_user(username)
-        except OSError as oe:
-            logger.error("Failed to clean up after account creation for {0}. {1}"
-                         .format(username, oe.strerror()))
         except Exception as e:
-            logger.error("Failed to clean up after account creation for {0}. {1}"
-                         .format(username, str(e)))
+            raise RemoteAccessError("Failed to clean up after account creation for {0}.".format(username), e)
+
+    def remove_user(self, username):
+        try:
+            self.delete_user(username)
+        except Exception as e:
+            raise RemoteAccessError("Failed to delete user {0}".format(username), e)
 
     def delete_user(self, username):
         self.os_util.del_account(username)
diff --git a/config/iosxe/waagent.conf b/config/iosxe/waagent.conf
new file mode 100644
index 00000000..1c36def6
--- /dev/null
+++ b/config/iosxe/waagent.conf
@@ -0,0 +1,119 @@
+#
+# Microsoft Azure Linux Agent Configuration
+#
+
+# Enable instance creation
+Provisioning.Enabled=n
+
+# Rely on cloud-init to provision
+Provisioning.UseCloudInit=n
+
+# Password authentication for root account will be unavailable.
+Provisioning.DeleteRootPassword=y
+
+# Generate fresh host key pair.
+Provisioning.RegenerateSshHostKeyPair=n
+
+# Supported values are "rsa", "dsa", "ecdsa", "ed25519", and "auto".
+# The "auto" option is supported on OpenSSH 5.9 (2011) and later.
+Provisioning.SshHostKeyPairType=rsa
+
+# Monitor host name changes and publish changes via DHCP requests.
+Provisioning.MonitorHostName=n
+
+# Decode CustomData from Base64.
+Provisioning.DecodeCustomData=n
+
+# Execute CustomData after provisioning.
+Provisioning.ExecuteCustomData=n
+
+# Algorithm used by crypt when generating password hash.
+#Provisioning.PasswordCryptId=6
+
+# Length of random salt used when generating password hash.
+#Provisioning.PasswordCryptSaltLength=10
+
+# Allow reset password of sys user
+Provisioning.AllowResetSysUser=n
+
+# Format if unformatted. If 'n', resource disk will not be mounted.
+ResourceDisk.Format=n
+
+# File system on the resource disk
+# Typically ext3 or ext4. FreeBSD images should use 'ufs2' here.
+ResourceDisk.Filesystem=ext4
+
+# Mount point for the resource disk
+ResourceDisk.MountPoint=/mnt/resource
+
+# Create and use swapfile on resource disk.
+ResourceDisk.EnableSwap=n
+
+# Size of the swapfile.
+ResourceDisk.SwapSizeMB=0
+
+# Comma-seperated list of mount options. See man(8) for valid options.
+ResourceDisk.MountOptions=None
+
+# Enable verbose logging (y|n)
+Logs.Verbose=n
+
+# Is FIPS enabled
+OS.EnableFIPS=n
+
+# Root device timeout in seconds.
+OS.RootDeviceScsiTimeout=300
+
+# If "None", the system default version is used.
+OS.OpensslPath=None
+
+# Set the SSH ClientAliveInterval
+# OS.SshClientAliveInterval=180
+
+# Set the path to SSH keys and configuration files
+OS.SshDir=/etc/ssh
+
+# If set, agent will use proxy server to access internet
+#HttpProxy.Host=None
+#HttpProxy.Port=None
+
+# Detect Scvmm environment, default is n
+# DetectScvmmEnv=n
+
+#
+# Lib.Dir=/var/lib/waagent
+
+#
+# DVD.MountPoint=/mnt/cdrom/secure
+
+#
+# Pid.File=/var/run/waagent.pid
+
+#
+# Extension.LogDir=/var/log/azure
+
+#
+# Home.Dir=/home
+
+# Enable RDMA management and set up, should only be used in HPC images
+# OS.EnableRDMA=y
+
+# Enable or disable goal state processing auto-update, default is enabled
+AutoUpdate.Enabled=y
+
+# Determine the update family, this should not be changed
+# AutoUpdate.GAFamily=Prod
+
+# Determine if the overprovisioning feature is enabled. If yes, hold extension
+# handling until inVMArtifactsProfile.OnHold is false.
+# Default is enabled
+# EnableOverProvisioning=y
+
+# Allow fallback to HTTP if HTTPS is unavailable
+# Note: Allowing HTTP (vs. HTTPS) may cause security risks
+# OS.AllowHTTP=n
+
+# Add firewall rules to protect access to Azure host node services
+# Note:
+# - The default is false to protect the state of existing VMs
+OS.EnableFirewall=y
diff --git a/setup.py b/setup.py
index 063779a8..e15a2d46 100755
--- a/setup.py
+++ b/setup.py
@@ -159,6 +159,15 @@ def get_data_files(name, version, fullname):
         set_conf_files(data_files, src=["config/debian/waagent.conf"])
         set_logrotate_files(data_files)
         set_udev_files(data_files, dest="/lib/udev/rules.d")
+    elif name == 'iosxe':
+        set_bin_files(data_files)
+        set_conf_files(data_files, src=["config/iosxe/waagent.conf"])
+        set_logrotate_files(data_files)
+        set_udev_files(data_files)
+        set_systemd_files(data_files, dest="/usr/lib/systemd/system")
+        if version.startswith("7.1"):
+            # TODO this is a mitigation to systemctl bug on 7.1
+            set_sysv_files(data_files)
     else:
         # Use default setting
         set_bin_files(data_files)
@@ -222,7 +231,7 @@ setuptools.setup(
     platforms='Linux',
     url='https://github.com/Azure/WALinuxAgent',
     license='Apache License Version 2.0',
-    packages=find_packages(exclude=["tests"]),
+    packages=find_packages(exclude=["tests*"]),
     py_modules=["__main__"],
     install_requires=requires,
     cmdclass={


Test patch:
diff --git a/tests/common/osutil/test_default.py b/tests/common/osutil/test_default.py
index b96b859c..653b0f30 100644
--- a/tests/common/osutil/test_default.py
+++ b/tests/common/osutil/test_default.py
@@ -189,9 +189,25 @@ class TestOSUtil(AgentTestCase):
             with patch.object(osutil.DefaultOSUtil, '_get_all_interfaces', return_value=fake_ifaces):
                 self.assertEqual(('', ''), osutil.DefaultOSUtil().get_first_if())
 
+    def test_get_all_interfaces(self):
+        loopback_count = 0
+        non_loopback_count = 0
+
+        for iface in osutil.DefaultOSUtil()._get_all_interfaces():
+            if iface == 'lo':
+                loopback_count += 1
+            else:
+                non_loopback_count += 1
+
+        self.assertEqual(loopback_count, 1, 'Exactly 1 loopback network interface should exist')
+        self.assertGreater(loopback_count, 0, 'At least 1 non-loopback network interface should exist')
+
     def test_isloopback(self):
-        self.assertTrue(osutil.DefaultOSUtil().is_loopback('lo'))
-        self.assertFalse(osutil.DefaultOSUtil().is_loopback('eth0'))
+        for iface in osutil.DefaultOSUtil()._get_all_interfaces():
+            if iface == 'lo':
+                self.assertTrue(osutil.DefaultOSUtil().is_loopback(iface))
+            else:
+                self.assertFalse(osutil.DefaultOSUtil().is_loopback(iface))
 
     def test_isprimary(self):
         routing_table = "\
diff --git a/tests/common/test_version.py b/tests/common/test_version.py
index d8c4ea93..bb7f4d37 100644
--- a/tests/common/test_version.py
+++ b/tests/common/test_version.py
@@ -23,10 +23,75 @@ import mock
 
 from azurelinuxagent.common.version import set_current_agent, \
     AGENT_LONG_VERSION, AGENT_VERSION, AGENT_NAME, AGENT_NAME_PATTERN, \
-    get_f5_platform
+    get_f5_platform, get_distro
 from tests.tools import *
 
 
+def freebsd_system():
+    return ["FreeBSD"]
+
+
+def freebsd_system_release(x, y, z):
+    return "10.0"
+
+
+def openbsd_system():
+    return ["OpenBSD"]
+
+
+def openbsd_system_release(x, y, z):
+    return "20.0"
+
+
+def default_system():
+    return [""]
+
+
+def default_system_no_linux_distro():
+    return '', '', ''
+
+def default_system_exception():
+    raise Exception
+
+
+class TestAgentVersion(AgentTestCase):
+    def setUp(self):
+        AgentTestCase.setUp(self)
+        return
+
+    @mock.patch('platform.system', side_effect=freebsd_system)
+    @mock.patch('re.sub', side_effect=freebsd_system_release)
+    def test_distro_is_correct_format_when_freebsd(self, platform_system_name, mock_variable):
+        osinfo = get_distro()
+        freebsd_list = ['freebsd', "10.0", '', 'freebsd']
+        self.assertListEqual(freebsd_list, osinfo)
+        return
+
+    @mock.patch('platform.system', side_effect=openbsd_system)
+    @mock.patch('re.sub', side_effect=openbsd_system_release)
+    def test_distro_is_correct_format_when_openbsd(self, platform_system_name, mock_variable):
+        osinfo = get_distro()
+        openbsd_list = ['openbsd', "20.0", '', 'openbsd']
+        self.assertListEqual(openbsd_list, osinfo)
+        return
+
+    @mock.patch('platform.system', side_effect=default_system)
+    @mock.patch('platform.dist', side_effect=default_system_no_linux_distro)
+    def test_distro_is_correct_format_when_default_case(self, platform_system_name, default_system_no_linux):
+        osinfo = get_distro()
+        default_list = ['', '', '', '']
+        self.assertListEqual(default_list, osinfo)
+        return
+
+    @mock.patch('platform.system', side_effect=default_system)
+    @mock.patch('platform.dist', side_effect=default_system_exception)
+    def test_distro_is_correct_for_exception_case(self, platform_system_name, default_system_no_linux):
+        osinfo = get_distro()
+        default_list = ['unknown', 'FFFF', '', '']
+        self.assertListEqual(default_list, osinfo)
+        return
+
+
 class TestCurrentAgentName(AgentTestCase):
     def setUp(self):
         AgentTestCase.setUp(self)
diff --git a/tests/ga/test_extension.py b/tests/ga/test_extension.py
index 7cb51668..9e6f3a44 100644
--- a/tests/ga/test_extension.py
+++ b/tests/ga/test_extension.py
@@ -326,7 +326,7 @@ class TestExtension(AgentTestCase):
         #Test hotfix
         test_data.goal_state = test_data.goal_state.replace("<Incarnation>2<",
                                                             "<Incarnation>3<")
-        test_data.ext_conf = test_data.ext_conf.replace("1.0.0", "1.1.0")
+        test_data.ext_conf = test_data.ext_conf.replace("1.0.0", "1.1.1")
         test_data.ext_conf = test_data.ext_conf.replace("seqNo=\"1\"", 
                                                         "seqNo=\"2\"")
         exthandlers_handler.run()
@@ -336,7 +336,7 @@ class TestExtension(AgentTestCase):
         #Test upgrade
         test_data.goal_state = test_data.goal_state.replace("<Incarnation>3<",
                                                             "<Incarnation>4<")
-        test_data.ext_conf = test_data.ext_conf.replace("1.1.0", "1.2.0")
+        test_data.ext_conf = test_data.ext_conf.replace("1.1.1", "1.2.0")
         test_data.ext_conf = test_data.ext_conf.replace("seqNo=\"2\"",
                                                         "seqNo=\"3\"")
         exthandlers_handler.run()
@@ -460,84 +460,72 @@ class TestExtension(AgentTestCase):
         self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.0.0")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
-        #Test goal state not changed
-        exthandlers_handler.run()
-        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.0.0")
-
-        #Test goal state changed without new GUID
+        #Test goal state changed
         test_data.goal_state = test_data.goal_state.replace("<Incarnation>1<",
                                                             "<Incarnation>2<")
         exthandlers_handler.run()
         self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.0.0")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
-        #Test GUID change without new version available
+        #Test minor version bump
         test_data.goal_state = test_data.goal_state.replace("<Incarnation>2<",
                                                             "<Incarnation>3<")
-        test_data.ext_conf = test_data.ext_conf.replace("FE0987654321", "FE0987654322")
+        test_data.ext_conf = test_data.ext_conf.replace("1.0.0", "1.1.0")
         exthandlers_handler.run()
-        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.0.0")
+        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.1.0")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
-        #Test hotfix available without GUID change
+        #Test hotfix version bump
         test_data.goal_state = test_data.goal_state.replace("<Incarnation>3<",
                                                             "<Incarnation>4<")
-        test_data.ext_conf = test_data.ext_conf.replace("1.0.0", "1.1.0")
-        exthandlers_handler.run()
-        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.0.0")
-        self._assert_ext_status(protocol.report_ext_status, "success", 0)
-
-        #Test GUID change with hotfix
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>4<",
-                                                            "<Incarnation>5<")
-        test_data.ext_conf = test_data.ext_conf.replace("FE0987654322", "FE0987654323")
+        test_data.ext_conf = test_data.ext_conf.replace("1.1.0", "1.1.1")
         exthandlers_handler.run()
         self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.1.1")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
         #Test disable
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>5<",
-                                                            "<Incarnation>6<")
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>4<",
+                                                            "<Incarnation>5<")
         test_data.ext_conf = test_data.ext_conf.replace("enabled", "disabled")
         exthandlers_handler.run()
         self._assert_handler_status(protocol.report_vm_status, "NotReady",
                                     1, "1.1.1")
 
         #Test uninstall
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>6<",
-                                                            "<Incarnation>7<")
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>5<",
+                                                            "<Incarnation>6<")
         test_data.ext_conf = test_data.ext_conf.replace("disabled", "uninstall")
         exthandlers_handler.run()
         self._assert_no_handler_status(protocol.report_vm_status)
 
         #Test uninstall again!
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>7<",
-                                                            "<Incarnation>8<")
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>6<",
+                                                            "<Incarnation>7<")
         exthandlers_handler.run()
         self._assert_no_handler_status(protocol.report_vm_status)
 
         #Test re-install
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>8<",
-                                                            "<Incarnation>9<")
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>7<",
+                                                            "<Incarnation>8<")
         test_data.ext_conf = test_data.ext_conf.replace("uninstall", "enabled")
         exthandlers_handler.run()
         self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.1.1")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
-        #Test upgrade available without GUID change
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>9<",
-                                                            "<Incarnation>10<")
-        test_data.ext_conf = test_data.ext_conf.replace("1.1.0", "1.2.0")
+        #Test version bump post-re-install
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>8<",
+                                                            "<Incarnation>9<")
+        test_data.ext_conf = test_data.ext_conf.replace("1.1.1", "1.2.0")
         exthandlers_handler.run()
-        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.1.1")
+        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.2.0")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
-        #Test GUID change with upgrade available
-        test_data.goal_state = test_data.goal_state.replace("<Incarnation>10<",
-                                                            "<Incarnation>11<")
-        test_data.ext_conf = test_data.ext_conf.replace("FE0987654323", "FE0987654324")
+        #Test rollback
+        test_data.goal_state = test_data.goal_state.replace("<Incarnation>9<",
+                                                            "<Incarnation>10<")
+        test_data.ext_conf = test_data.ext_conf.replace("1.2.0", "1.1.0")
         exthandlers_handler.run()
-        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.2.0")
+        self._assert_handler_status(protocol.report_vm_status, "Ready", 1, "1.1.0")
         self._assert_ext_status(protocol.report_ext_status, "success", 0)
 
     @patch('azurelinuxagent.ga.exthandlers.add_event')
@@ -669,12 +657,11 @@ class TestExtension(AgentTestCase):
                         datafile = DATA_FILE_EXT_INTERNALVERSION
                 else:
                     config_version = '1.0.0'
+                    decision_version = '1.0.0'
                     if autoupgrade:
                         datafile = DATA_FILE_EXT_AUTOUPGRADE
-                        decision_version = '1.2.0'
                     else:
                         datafile = DATA_FILE
-                        decision_version = '1.0.0'
 
                 _, protocol = self._create_mock(WireProtocolData(datafile), *args)
                 ext_handlers, _ = protocol.get_ext_handlers()
@@ -695,22 +682,25 @@ class TestExtension(AgentTestCase):
 
         # (installed_version, config_version, exptected_version, autoupgrade_expected_version)
         cases = [
-            (None,  '2.0',     '2.0.0',    '2.2.0'),
-            (None,  '2.0.0',   '2.0.0',    '2.2.0'),
-            ('1.0', '1.0.0',   '1.0.0',    '1.2.0'),
-            (None,  '2.1.0',   '2.1.1',    '2.2.0'),
-            (None,  '2.2.0',   '2.2.0',    '2.2.0'),
-            (None,  '2.3.0',   '2.3.0',    '2.3.0'),
-            (None,  '2.4.0',   '2.4.0',    '2.4.0'),
-            (None,  '3.0',     '3.0',      '3.1'),
-            (None,  '4.0',     '4.0.0.1',  '4.1.0.0'),
+            (None,  '2.0',     '2.0.0'),
+            (None,  '2.0.0',   '2.0.0'),
+            ('1.0', '1.0.0',   '1.0.0'),
+            (None,  '2.1.0',   '2.1.0'),
+            (None,  '2.1.1',   '2.1.1'),
+            (None,  '2.2.0',   '2.2.0'),
+            (None,  '2.3.0',   '2.3.0'),
+            (None,  '2.4.0',   '2.4.0'),
+            (None,  '3.0',     '3.0'),
+            (None,  '3.1',     '3.1'),
+            (None,  '4.0',     '4.0.0.1'),
+            (None,  '4.1',     '4.1.0.0'),
         ]
 
         _, protocol = self._create_mock(WireProtocolData(DATA_FILE), *args)
         version_uri = Mock()
         version_uri.uri = 'http://some/Microsoft.OSTCExtensions_ExampleHandlerLinux_asiaeast_manifest.xml'
 
-        for (installed_version, config_version, expected_version, autoupgrade_expected_version) in cases:
+        for (installed_version, config_version, expected_version) in cases:
             ext_handler = Mock()
             ext_handler.properties = Mock()
             ext_handler.name = 'OSTCExtensions.ExampleHandlerLinux'
@@ -723,15 +713,6 @@ class TestExtension(AgentTestCase):
             ext_handler_instance.decide_version()
             self.assertEqual(expected_version, ext_handler.properties.version)
 
-            ext_handler.properties.version = config_version
-            ext_handler.properties.upgradePolicy = 'auto'
-
-            ext_handler_instance = ExtHandlerInstance(ext_handler, protocol)
-            ext_handler_instance.get_installed_version = Mock(return_value=installed_version)
-
-            ext_handler_instance.decide_version()
-            self.assertEqual(autoupgrade_expected_version, ext_handler.properties.version)
-
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/tests/ga/test_remoteaccess_handler.py b/tests/ga/test_remoteaccess_handler.py
index d0147351..71271c97 100644
--- a/tests/ga/test_remoteaccess_handler.py
+++ b/tests/ga/test_remoteaccess_handler.py
@@ -16,6 +16,8 @@
 #
 
 from datetime import timedelta
+
+from azurelinuxagent.common.exception import RemoteAccessError
 from azurelinuxagent.common.protocol.wire import *
 from azurelinuxagent.ga.remoteaccess import RemoteAccessHandler
 from tests.common.osutil.mock_osutil import MockOSUtil
@@ -41,12 +43,19 @@ def log_error(msg_format, *args):
     error_messages.append(msg_format.format(args))
 
 
+def mock_add_event(name, op, is_success, version, message):
+    TestRemoteAccessHandler.eventing_data = (name, op, is_success, version, message)
+
+
 class TestRemoteAccessHandler(AgentTestCase):
+    eventing_data = [()]
 
     def setUp(self):
         super(TestRemoteAccessHandler, self).setUp()
         del info_messages[:]
         del error_messages[:]
+        for data in TestRemoteAccessHandler.eventing_data:
+            del data
 
     # add_user tests
     @patch('azurelinuxagent.common.logger.Logger.info', side_effect=log_info)
@@ -79,14 +88,13 @@ class TestRemoteAccessHandler(AgentTestCase):
         rah.os_util = MockOSUtil()
         tstpassword = "]aPPEv}uNg1FPnl?"
         tstuser = ""
-        expiration_date = datetime.utcnow() + timedelta(days=1)
+        expiration = datetime.utcnow() + timedelta(days=1)
         pwd = tstpassword
-        rah.add_user(tstuser, pwd, expiration_date)
+        error = "Error adding user {0}. test exception for bad username".format(tstuser)
+        self.assertRaisesRegex(RemoteAccessError, error, rah.add_user, tstuser, pwd, expiration)
         self.assertEqual(0, len(rah.os_util.get_users()))
-        self.assertEqual(1, len(error_messages))
+        self.assertEqual(0, len(error_messages))
         self.assertEqual(0, len(info_messages))
-        error = "Error adding user {0}. test exception for bad username".format(tstuser)
-        self.assertEqual(error, error_messages[0])
 
     @patch('azurelinuxagent.common.logger.Logger.info', side_effect=log_info)
     @patch('azurelinuxagent.common.logger.Logger.error', side_effect=log_error)
@@ -96,16 +104,16 @@ class TestRemoteAccessHandler(AgentTestCase):
         rah.os_util = MockOSUtil()
         tstpassword = ""
         tstuser = "foobar"
-        expiration_date = datetime.utcnow() + timedelta(days=1)
+        expiration = datetime.utcnow() + timedelta(days=1)
         pwd = tstpassword
-        rah.add_user(tstuser, pwd, expiration_date)
+        error = "Error adding user {0} cleanup successful\nInner error: test exception for bad password".format(tstuser)
+        self.assertRaisesRegex(RemoteAccessError, error, rah.add_user, tstuser, pwd, expiration)
         self.assertEqual(0, len(rah.os_util.get_users()))
-        self.assertEqual(1, len(error_messages))
+        self.assertEqual(0, len(error_messages))
         self.assertEqual(1, len(info_messages))
-        error = "Error creating user {0}. test exception for bad password".format(tstuser)
-        self.assertEqual(error, error_messages[0])
         self.assertEqual("User deleted {0}".format(tstuser), info_messages[0])
 
+
     @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
            return_value="]aPPEv}uNg1FPnl?")
     def test_add_user_already_existing(self, _):
@@ -125,7 +133,7 @@ class TestRemoteAccessHandler(AgentTestCase):
         # this does not test the user add function as that's mocked, it tests processing skips the remaining
         # calls after the initial failure
         new_user_expiration = datetime.utcnow() + timedelta(days=5)
-        rah.add_user(tstuser, pwd, new_user_expiration)
+        self.assertRaises(RemoteAccessError, rah.add_user, tstuser, pwd, new_user_expiration)
         # refresh users
         users = get_user_dictionary(rah.os_util.get_users())
         self.assertTrue(tstuser in users, "{0} missing from users after dup user attempted".format(tstuser))
@@ -164,7 +172,7 @@ class TestRemoteAccessHandler(AgentTestCase):
         mock_os_util.all_users[testusr] = (testusr, None, None, None, None, None, None, None)
         rah = RemoteAccessHandler()
         rah.os_util = mock_os_util
-        rah.handle_failed_create("", "test message")
+        self.assertRaises(RemoteAccessError, rah.handle_failed_create, "")
         users = get_user_dictionary(rah.os_util.get_users())
         self.assertEqual(1, len(users.keys()))
         self.assertTrue(testusr in users, "Expected user {0} missing".format(testusr))
@@ -178,17 +186,14 @@ class TestRemoteAccessHandler(AgentTestCase):
         rah = RemoteAccessHandler()
         rah.os_util = mock_os_util
         testuser = "Carl"
-        test_message = "test message"
-        rah.handle_failed_create(testuser, test_message)
+        error = "Failed to clean up after account creation for {0}.\n" \
+                "Inner error: test exception, user does not exist to delete".format(testuser)
+        self.assertRaisesRegex(RemoteAccessError, error, rah.handle_failed_create, testuser)
         users = get_user_dictionary(rah.os_util.get_users())
         self.assertEqual(1, len(users.keys()))
         self.assertTrue(testusr in users, "Expected user {0} missing".format(testusr))
-        self.assertEqual(2, len(error_messages))
+        self.assertEqual(0, len(error_messages))
         self.assertEqual(0, len(info_messages))
-        self.assertEqual("Error creating user {0}. {1}".format(testuser, test_message), error_messages[0])
-        msg = "Failed to clean up after account creation for {0}. test exception, user does not exist to delete"\
-            .format(testuser)
-        self.assertEqual(msg, error_messages[1])
 
     @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
            return_value="]aPPEv}uNg1FPnl?")
@@ -230,14 +235,14 @@ class TestRemoteAccessHandler(AgentTestCase):
         tstuser = "foobar"
         expiration = datetime.utcnow() + timedelta(days=1)
         pwd = "bad password"
-        rah.add_user(tstuser, pwd, expiration)
+        error = "Error adding user foobar cleanup successful\n" \
+                "Inner error: \[CryptError\] Error decoding secret\n" \
+                "Inner error: Incorrect padding".format(tstuser)
+        self.assertRaisesRegex(RemoteAccessError, error, rah.add_user, tstuser, pwd, expiration)
         users = get_user_dictionary(rah.os_util.get_users())
         self.assertEqual(0, len(users))
-        self.assertEqual(1, len(error_messages))
+        self.assertEqual(0, len(error_messages))
         self.assertEqual(1, len(info_messages))
-        error = "Error creating user {0}. [CryptError] Error decoding secret\nInner error: Incorrect padding".\
-            format(tstuser)
-        self.assertEqual(error, error_messages[0])
         self.assertEqual("User deleted {0}".format(tstuser), info_messages[0])
 
     def test_handle_remote_access_no_users(self):
@@ -395,3 +400,128 @@ class TestRemoteAccessHandler(AgentTestCase):
         self.assertTrue(tstuser in users, "{0} missing from users".format(tstuser))
         rah.run()
         self.assertTrue(tstuser in users, "{0} missing from users".format(tstuser))
+
+    @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
+           return_value="]aPPEv}uNg1FPnl?")
+    def test_handle_remote_access_multiple_users_one_removed(self, _):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        data_str = load_data('wire/remote_access_10_accounts.xml')
+        remote_access = RemoteAccess(data_str)
+        count = 0
+        for user in remote_access.user_list.users:
+            count += 1
+            user.name = "tstuser{0}".format(count)
+            expiration_date = datetime.utcnow() + timedelta(days=count)
+            user.expiration = expiration_date.strftime("%a, %d %b %Y %H:%M:%S ") + "UTC"
+        rah.remote_access = remote_access
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(10, len(users))
+        # now remove the user from RemoteAccess
+        deleted_user = rah.remote_access.user_list.users[3]
+        del rah.remote_access.user_list.users[3]
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertTrue(deleted_user not in users, "{0} still in users".format(deleted_user))
+        self.assertEqual(9, len(users))
+
+    @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
+           return_value="]aPPEv}uNg1FPnl?")
+    def test_handle_remote_access_multiple_users_null_remote_access(self, _):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        data_str = load_data('wire/remote_access_10_accounts.xml')
+        remote_access = RemoteAccess(data_str)
+        count = 0
+        for user in remote_access.user_list.users:
+            count += 1
+            user.name = "tstuser{0}".format(count)
+            expiration_date = datetime.utcnow() + timedelta(days=count)
+            user.expiration = expiration_date.strftime("%a, %d %b %Y %H:%M:%S ") + "UTC"
+        rah.remote_access = remote_access
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(10, len(users))
+        # now remove the user from RemoteAccess
+        rah.remote_access = None
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(0, len(users))
+
+    @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
+           return_value="]aPPEv}uNg1FPnl?")
+    def test_handle_remote_access_multiple_users_error_with_null_remote_access(self, _):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        data_str = load_data('wire/remote_access_10_accounts.xml')
+        remote_access = RemoteAccess(data_str)
+        count = 0
+        for user in remote_access.user_list.users:
+            count += 1
+            user.name = "tstuser{0}".format(count)
+            expiration_date = datetime.utcnow() + timedelta(days=count)
+            user.expiration = expiration_date.strftime("%a, %d %b %Y %H:%M:%S ") + "UTC"
+        rah.remote_access = remote_access
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(10, len(users))
+        # now remove the user from RemoteAccess
+        rah.remote_access = None
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(0, len(users))
+
+    def test_remove_user_error(self):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        error = "Failed to delete user {0}\nInner error: test exception, bad data".format("")
+        self.assertRaisesRegex(RemoteAccessError, error, rah.remove_user, "")
+
+    def test_remove_user_not_exists(self):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        user = "bob"
+        error = "Failed to delete user {0}\n" \
+                "Inner error: test exception, user does not exist to delete".format(user)
+        self.assertRaisesRegex(RemoteAccessError, error, rah.remove_user, user)
+
+    @patch('azurelinuxagent.common.utils.cryptutil.CryptUtil.decrypt_secret',
+           return_value="]aPPEv}uNg1FPnl?")
+    def test_handle_remote_access_remove_and_add(self, _):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        data_str = load_data('wire/remote_access_10_accounts.xml')
+        remote_access = RemoteAccess(data_str)
+        count = 0
+        for user in remote_access.user_list.users:
+            count += 1
+            user.name = "tstuser{0}".format(count)
+            expiration_date = datetime.utcnow() + timedelta(days=count)
+            user.expiration = expiration_date.strftime("%a, %d %b %Y %H:%M:%S ") + "UTC"
+        rah.remote_access = remote_access
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertEqual(10, len(users))
+        # now remove the user from RemoteAccess
+        new_user = "tstuser11"
+        deleted_user = rah.remote_access.user_list.users[3]
+        rah.remote_access.user_list.users[3].name = new_user
+        rah.handle_remote_access()
+        users = rah.os_util.get_users()
+        self.assertTrue(deleted_user not in users, "{0} still in users".format(deleted_user))
+        self.assertTrue(new_user in [u[0] for u in users], "user {0} not in users".format(new_user))
+        self.assertEqual(10, len(users))
+
+    @patch('azurelinuxagent.ga.remoteaccess.add_event', side_effect=mock_add_event)
+    @patch('azurelinuxagent.common.protocol.util.ProtocolUtil.get_protocol', side_effect=RemoteAccessError("foobar!"))
+    def test_remote_access_handler_run_error(self, _1, _2):
+        rah = RemoteAccessHandler()
+        rah.os_util = MockOSUtil()
+        rah.run()
+        print(TestRemoteAccessHandler.eventing_data)
+        check_message = "foobar!"
+        self.assertTrue(check_message in TestRemoteAccessHandler.eventing_data[4],
+                        "expected message {0} not found in {1}"
+                        .format(check_message, TestRemoteAccessHandler.eventing_data[4]))
+        self.assertEqual(False, TestRemoteAccessHandler.eventing_data[2], "is_success is true")
diff --git a/tests/tools.py b/tests/tools.py
index 9de73778..22278e31 100644
--- a/tests/tools.py
+++ b/tests/tools.py
@@ -18,8 +18,9 @@
 """
 Define util functions for unit test
 """
-
+import difflib
 import os
+import pprint
 import re
 import shutil
 import tempfile
@@ -60,6 +61,17 @@ def _do_nothing():
 
 _MAX_LENGTH = 120
 
+_MAX_LENGTH_SAFE_REPR = 80
+
+def safe_repr(obj, short=False):
+    try:
+        result = repr(obj)
+    except Exception:
+        result = object.__repr__(obj)
+    if not short or len(result) < _MAX_LENGTH:
+        return result
+    return result[:_MAX_LENGTH_SAFE_REPR] + ' [truncated]...'
+
 
 def skip_if_predicate_false(predicate, message):
     if not predicate():
@@ -111,6 +123,12 @@ class AgentTestCase(unittest.TestCase):
             cls.assertIsNone = cls.emulate_assertIsNone
         if not hasattr(cls, "assertIsNotNone"):
             cls.assertIsNone = cls.emulate_assertIsNotNone
+        if hasattr(cls, "assertRaisesRegexp"):
+            cls.assertRaisesRegex = cls.assertRaisesRegexp
+        if not hasattr(cls, "assertRaisesRegex"):
+            cls.assertRaisesRegex = cls.emulate_raises_regex
+        if not hasattr(cls, "assertListEqual"):
+            cls.assertListEqual = cls.emulate_assertListEqual
 
     def setUp(self):
         prefix = "{0}_".format(self.__class__.__name__)
@@ -163,6 +181,121 @@ class AgentTestCase(unittest.TestCase):
             msg = msg if msg is not None else '{0} is None'.format(_safe_repr(x))
             self.fail(msg)
 
+
+    def emulate_raises_regex(self, exception_type, regex, function, *args, **kwargs):
+        try:
+            function(*args, **kwargs)
+        except Exception as e:
+            if re.search(regex, str(e), flags=1) is not None:
+                return
+            else:
+                self.fail("Expected exception {0} matching {1}.  Actual: {2}".format(
+                    exception_type, regex, str(e)))
+        self.fail("No exception was thrown.  Expected exception {0} matching {1}".format(exception_type, regex))
+
+    def emulate_assertListEqual(self, seq1, seq2, msg=None, seq_type=None):
+        """An equality assertion for ordered sequences (like lists and tuples).
+
+        For the purposes of this function, a valid ordered sequence type is one
+        which can be indexed, has a length, and has an equality operator.
+
+        Args:
+            seq1: The first sequence to compare.
+            seq2: The second sequence to compare.
+            seq_type: The expected datatype of the sequences, or None if no
+                    datatype should be enforced.
+            msg: Optional message to use on failure instead of a list of
+                    differences.
+        """
+        if seq_type is not None:
+            seq_type_name = seq_type.__name__
+            if not isinstance(seq1, seq_type):
+                raise self.failureException('First sequence is not a %s: %s'
+                                        % (seq_type_name, safe_repr(seq1)))
+            if not isinstance(seq2, seq_type):
+                raise self.failureException('Second sequence is not a %s: %s'
+                                        % (seq_type_name, safe_repr(seq2)))
+        else:
+            seq_type_name = "sequence"
+
+        differing = None
+        try:
+            len1 = len(seq1)
+        except (TypeError, NotImplementedError):
+            differing = 'First %s has no length.    Non-sequence?' % (
+                    seq_type_name)
+
+        if differing is None:
+            try:
+                len2 = len(seq2)
+            except (TypeError, NotImplementedError):
+                differing = 'Second %s has no length.    Non-sequence?' % (
+                        seq_type_name)
+
+        if differing is None:
+            if seq1 == seq2:
+                return
+
+            seq1_repr = safe_repr(seq1)
+            seq2_repr = safe_repr(seq2)
+            if len(seq1_repr) > 30:
+                seq1_repr = seq1_repr[:30] + '...'
+            if len(seq2_repr) > 30:
+                seq2_repr = seq2_repr[:30] + '...'
+            elements = (seq_type_name.capitalize(), seq1_repr, seq2_repr)
+            differing = '%ss differ: %s != %s\n' % elements
+
+            for i in xrange(min(len1, len2)):
+                try:
+                    item1 = seq1[i]
+                except (TypeError, IndexError, NotImplementedError):
+                    differing += ('\nUnable to index element %d of first %s\n' %
+                                 (i, seq_type_name))
+                    break
+
+                try:
+                    item2 = seq2[i]
+                except (TypeError, IndexError, NotImplementedError):
+                    differing += ('\nUnable to index element %d of second %s\n' %
+                                 (i, seq_type_name))
+                    break
+
+                if item1 != item2:
+                    differing += ('\nFirst differing element %d:\n%s\n%s\n' %
+                                 (i, safe_repr(item1), safe_repr(item2)))
+                    break
+            else:
+                if (len1 == len2 and seq_type is None and
+                    type(seq1) != type(seq2)):
+                    # The sequences are the same, but have differing types.
+                    return
+
+            if len1 > len2:
+                differing += ('\nFirst %s contains %d additional '
+                             'elements.\n' % (seq_type_name, len1 - len2))
+                try:
+                    differing += ('First extra element %d:\n%s\n' %
+                                  (len2, safe_repr(seq1[len2])))
+                except (TypeError, IndexError, NotImplementedError):
+                    differing += ('Unable to index element %d '
+                                  'of first %s\n' % (len2, seq_type_name))
+            elif len1 < len2:
+                differing += ('\nSecond %s contains %d additional '
+                             'elements.\n' % (seq_type_name, len2 - len1))
+                try:
+                    differing += ('First extra element %d:\n%s\n' %
+                                  (len1, safe_repr(seq2[len1])))
+                except (TypeError, IndexError, NotImplementedError):
+                    differing += ('Unable to index element %d '
+                                  'of second %s\n' % (len1, seq_type_name))
+        standardMsg = differing
+        diffMsg = '\n' + '\n'.join(
+            difflib.ndiff(pprint.pformat(seq1).splitlines(),
+                          pprint.pformat(seq2).splitlines()))
+        standardMsg = self._truncateMessage(standardMsg, diffMsg)
+        msg = self._formatMessage(msg, standardMsg)
+        self.fail(msg)
+
     @staticmethod
     def _create_files(tmp_dir, prefix, suffix, count, with_sleep=0):
         for i in range(count):
diff --git a/tests/utils/test_rest_util.py b/tests/utils/test_rest_util.py
index 05911f82..76b79479 100644
--- a/tests/utils/test_rest_util.py
+++ b/tests/utils/test_rest_util.py
@@ -25,7 +25,6 @@ from azurelinuxagent.common.future import httpclient, ustr
 
 from tests.tools import *
 
-
 class TestIOErrorCounter(AgentTestCase):
     def test_increment_hostplugin(self):
         restutil.IOErrorCounter.reset()
@@ -86,6 +85,7 @@ class TestIOErrorCounter(AgentTestCase):
            {"hostplugin":0, "protocol":0, "other":0},
             restutil.IOErrorCounter._counts)
 
+
 class TestHttpOperations(AgentTestCase):
     def test_parse_url(self):
         test_uri = "http://abc.def/ghi#hash?jkl=mn"
@@ -115,6 +115,68 @@ class TestHttpOperations(AgentTestCase):
         self.assertEquals(None, host)
         self.assertEquals(rel_uri, "None")
 
+    def test_cleanup_sas_tokens_from_urls_for_normal_cases(self):
+        test_url = "http://abc.def/ghi#hash?jkl=mn"
+        filtered_url = restutil.redact_sas_tokens_in_urls(test_url)
+        self.assertEquals(test_url, filtered_url)
+
+        test_url = "http://abc.def:80/"
+        filtered_url = restutil.redact_sas_tokens_in_urls(test_url)
+        self.assertEquals(test_url, filtered_url)
+
+        test_url = "http://abc.def/"
+        filtered_url = restutil.redact_sas_tokens_in_urls(test_url)
+        self.assertEquals(test_url, filtered_url)
+
+        test_url = "https://abc.def/ghi?jkl=mn"
+        filtered_url = restutil.redact_sas_tokens_in_urls(test_url)
+        self.assertEquals(test_url, filtered_url)
+
+    def test_cleanup_sas_tokens_from_urls_containing_sas_tokens(self):
+        # Contains pair of URLs (RawURL, RedactedURL)
+        urls_tuples = [("https://abc.def.xyz.123.net/functiontest/yokawasa.png?sig"
+                        "=sXBjML1Fpk9UnTBtajo05ZTFSk0LWFGvARZ6WlVcAog%3D&srt=o&ss=b&"
+                        "spr=https&sp=rl&sv=2016-05-31&se=2017-07-01T00%3A21%3A38Z&"
+                        "st=2017-07-01T23%3A16%3A38Z",
+                        "https://abc.def.xyz.123.net/functiontest/yokawasa.png?sig"
+                        "=" + restutil.REDACTED_TEXT +
+                        "&srt=o&ss=b&spr=https&sp=rl&sv=2016-05-31&se=2017-07-01T00"
+                        "%3A21%3A38Z&st=2017-07-01T23%3A16%3A38Z"),
+                       ("https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se=2018-07"
+                        "-26T02:20:44Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=DavQgRtl99DsEPv9Xeb63GnLXCuaLYw5ay%2BE1cFckQY%3D",
+                        "https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se"
+                        "=2018-07-26T02:20:44Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=" + restutil.REDACTED_TEXT),
+                       ("https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se=2018-07"
+                        "-26T02:20:44Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=ttSCKmyjiDEeIzT9q7HtYYgbCRIXuesFSOhNEab52NM%3D",
+                        "https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se"
+                        "=2018-07-26T02:20:44Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=" + restutil.REDACTED_TEXT),
+                       ("https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se=2018-07"
+                        "-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=X0imGmcj5KcBPFcqlfYjIZakzGrzONGbRv5JMOnGrwc%3D",
+                        "https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se"
+                        "=2018-07-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=" + restutil.REDACTED_TEXT),
+                       ("https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se=2018-07"
+                        "-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=9hfxYvaZzrMahtGO1OgMUiFGnDOtZXulZ3skkv1eVBg%3D",
+                        "https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se"
+                        "=2018-07-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https,"
+                        "http&sig=" + restutil.REDACTED_TEXT),
+                       ("https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se=2018-07"
+                        "-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https"
+                        "&sig=cmluQEHnOGsVK9NDm83ruuPdPWNQcerfjOAbkspNZXU%3D",
+                        "https://abc.def.xyz.123.net/?sv=2017-11-09&ss=b&srt=o&sp=r&se"
+                        "=2018-07-26T02:20:42Z&st=2018-07-25T18:20:44Z&spr=https&sig"
+                        "=" + restutil.REDACTED_TEXT)
+                       ]
+
+        for x in urls_tuples:
+            self.assertEquals(restutil.redact_sas_tokens_in_urls(x[0]), x[1])
+
     @patch('azurelinuxagent.common.conf.get_httpproxy_port')
     @patch('azurelinuxagent.common.conf.get_httpproxy_host')
     def test_get_http_proxy_none_is_default(self, mock_host, mock_port):