Repository: Azure/WALinuxAgent
Pull request #259 (2016-06-23 00:15:34)

Update RDMA IB interface from SharedConfig
This change ports https://github.com/Azure/WALinuxAgent/blob/adfb590ad3904e261da5605b36c3924afa22b2c5/waagent#L4239-L4324 and https://github.com/Azure/WALinuxAgent/blob/adfb590ad3904e261da5605b36c3924afa22b2c5/waagent#L4197-L4203 of v2.0 code to v2.1.

The code flow did not change. In the GoalState parsing, we parse SharedConfig for the first time we and spin up a thread to:

* parse `rdmaMacAddress`/`rdmaIPv4Address` from SharedConfig.xml
* update `dat.conf` with rdma IPv4 addr
* wait for `/dev/hvnd_rdma`
* write IPv4/MAC addrs to `/dev/hvnd_rdma`
* find network interface name for infiniband
* `ifconfig up` && `ifconfig [...] $IPv4Addr/16`

On errors, we throw exceptions but we catch them, so we do not crash the agent (keeping the current behavior).

Mostly manually tested and it seems to be working fine on CentOS.

**The only problem is** it looks like we run the agent twice:
```
  ├─python -u /usr/sbin/waagent -daemon
  │   ├─python /usr/sbin/waagent run-exthandlers
```

and if I add this change to GoalState processing, the code is being executed twice in separate processes. How do we prevent this? (Ideally this should be done in `daemon` and not `exthandlers`)

@rjschwei I encourage you to pick this up and test it on  SUSE. (although the issue above should be fixed, the code is testable.) I also appreciate the code reviews!

Here's how logs should look like (in journalctl, waagent.log will have duplications due to issue above):

```diff


Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.678873 INFO Azure Linux Agent Version:2.1.5.rc0
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.680079 INFO OS: centos 7.1.1503
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.680768 INFO Python: 2.7.5
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.681222 INFO Run daemon
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.681819 ERROR RDMAHandler.install_driver not implemented
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.682168 INFO Activate resource disk
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.688779 INFO Resource disk /dev/sdb1 is already mounted
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.689844 INFO Clean protocol
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.690377 INFO Start env monitor service.
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.692337 INFO Configure routes
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.692666 INFO Gateway:None
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.693016 INFO Routes:None
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.707830 INFO Detect protocol endpoints
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.709284 INFO Clean protocol
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.710388 INFO WireServer endpoint is not found. Rerun dhcp handler
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.711386 INFO test for route to 168.63.129.16
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.714847 INFO route to 168.63.129.16 exists
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.715688 INFO Wire server endpoint:168.63.129.16
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.720836 INFO Fabric preferred wire protocol version:2015-04-05
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.723281 INFO Wire protocol version:2012-11-30
Jun 23 00:08:38 centos python[49540]: 2016/06/23 00:08:38.724916 WARNING Server prefered version:2015-04-05
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.850724 INFO RDMA capabilities are enabled in configuration
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.852225 INFO Found RDMA details. IPv4=172.16.1.20 MAC=00:15:5D:33:FF:1D
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.852406 INFO RDMA: starting device processing in the background.
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.852936 INFO Updating DAPL configuration file
Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.853899 INFO Event: name=WALA, op=HeartBeat, message=
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.854191 INFO RDMA: trying /etc/dat.conf
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.855335 INFO RDMA: DAPL config not found at /etc/dat.conf
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.856497 INFO RDMA: trying /etc/rdma/dat.conf
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.857421 INFO RDMA: DAPL config is at: /etc/rdma/dat.conf
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.858466 INFO RDMA: DAPL configuration is updated
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.859412 INFO RDMA: waiting for device /dev/hvnd_rdma
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.859845 INFO RDMA: device ready
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.860166 INFO RDMA: Updating device with configuration: rdmaMacAddress="00:15:5D:33:FF:1D" rdmaIPv4Address="172.16.1.20"
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.864575 INFO RDMA: Updated device with IPv4/MAC addr successfully
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.864915 INFO RDMA: will update the network interface with IPv4/MAC
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.869903 INFO RDMA: network interface found: eth1
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.870169 INFO RDMA: bringing network interface up
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.873663 INFO RDMA: configuring IPv4 addr and netmask on interface
+Jun 23 00:08:42 centos python[49540]: 2016/06/23 00:08:42.876738 INFO RDMA: network address and netmask configured on interface
```


msftclas: Hi __@ahmetalpbalkan__, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
    <p>
        It looks like you're a Microsoft contributor (Ahmet Alp Balkan). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.
    </p>

TTYL, MSBOT;

rjschwei: Tested.

The code does configure the additional network:

~> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:3a:10:01:3b brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.5/24 brd 10.2.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::20d:3aff:fe10:13b/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:15:5d:33:ff:35 brd ff:ff:ff:ff:ff:ff
    inet 172.16.1.44/16 brd 172.16.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::215:5dff:fe33:ff35/64 scope link 
       valid_lft forever preferred_lft forever

The driver is loaded as expected:

~> lsmod | grep hv
hv_network_direct     109186  0 
ib_core                96404  11 rdma_cm,ib_cm,ib_sa,iw_cm,ib_mad,ib_ucm,ib_umad,hv_network_direct,ib_uverbs,rdma_ucm,ib_ipoib
ib_addr                18923  4 rdma_cm,ib_core,hv_network_direct,rdma_ucm
hv_netvsc              39767  0 
hv_utils               28477  0 
hv_storvsc             22171  2 
hv_vmbus              544328  7 hyperv_keyboard,hv_netvsc,hid_hyperv,hv_utils,hyperv_fb,hv_storvsc,hv_network_direct
scsi_mod              244624  5 sg,libata,sd_mod,sr_mod,hv_storvsc

Thus things appear to work as expected.
ahmetalpbalkan: Pushed some more commits fixing code review comments.
ahmetalpbalkan: > The only problem is it looks like we run the agent twice (-daemon and -exthandlers) and if I add this change to GoalState processing, the code is being executed twice in separate processes. How do we prevent this?

@hglkrijger @brendandixon I need help with this problem above and code reviewing this.


brendandixon: @ahmetalpbalkan The path is as follows:

Parent process:
- Agent#main launches the daemon
- daemon invokes UpdateHandler#run_latest
- UpdateHandler#run_latest spawns the latest agent with "run-exthandlers"

Child process:
- Agent#main launches UpdateHandler#run
- UpdateHandler#run processes goal state

I am confused. Where are you placing the code such that it's executed twice? In the above model, it should occur only once.
ahmetalpbalkan: @brendandixon code is at `update_goal_state` function, which is supposed to update files on disk with  Goal State retrieved from file server. This method is simply called from both child & parent processes (otherwise my code wouldn't be invoked twice).

This also means that the XML files we write on disk for diagnostics are subject to race condition or corruption as well.
brendandixon: @ahmetalpbalkan That placement feels incorrect. It's making RDMA handling a side-effect of other work. Why not raise it to a top-level method called just after provisioning completes?
brendandixon: @ahmetalpbalkan And, no, there is no race condition because of the ordering I described above. While there are multiple processes, their ordering is very clean.
ahmetalpbalkan: @brendandixon Goal State is retrieved after the provisioning is completed. Where else can I happily put my code where I have access to the `SharedConfig.xml` contents and get invoked from only one process?
ahmetalpbalkan: @brendandixon and I'm afraid you're wrong, there is indeed a race condition. SharedConfig.xml is written to the disk **by two processes** within 200ms.

2016/06/24 19:46:14.043846 INFO -------------> WRITE TO SHAREDCONFIG.XML pid=57858
2016/06/24 19:46:14.200146 INFO -------------> WRITE TO SHAREDCONFIG.XML pid=57873

just edit `azurelinuxagent/common/protocol/wire.py` and add this:

```diff
    def update_shared_conf(self, goal_state):
+        import os
+        logger.info('-------------> WRITE TO SHAREDCONFIG.XML pid=%d' % os.getpid())
```
hglkrijger: apart from the one comment, looks good to me
ahmetalpbalkan: @brendandixon moved the RDMA device setup logic outside the goal state loop. We now look at the SharedConfig.xml file on the disk (written by the processes earlier) to parse the info and use it to setup the RDMA before the main loop.
brendandixon: @ahmetalpbalkan What guarantee does the code have that SharedConfig.xml will be written beforehand? Is it a side-effect of earlier provisioning code?
ahmetalpbalkan: @brendandixon based on my observation, the SharedeConfig.xml gets written by the same (parent) process earlier to the disk. I think it is a side-effect of something (probably not the provisioning?).
brendandixon: @ahmetalpbalkan I'm concerned that it's written by the monitor.py code...where we just removed the side-effect. If you absolutely have to have a SharedConfig.xml file, then, *maybe*, the code should reside inside the UpdateHandler#run loop (as much as I hate to suggest it...).
ahmetalpbalkan: @brendandixon hmm I rebased the code on top of your fix, and SharedConfig.xml was still being written. Do you want me to move it back to the GoalState handling?
brendandixon: @ahmetalpbalkan No. If you've rebased and it's working (on a clean machine without residual files), then let's keep it as is.
ahmetalpbalkan: @brendandixon I will test again just to make sure.

Patch:
diff --git a/azurelinuxagent/common/dhcp.py b/azurelinuxagent/common/dhcp.py
index 8c81f248..d5c90cb0 100644
--- a/azurelinuxagent/common/dhcp.py
+++ b/azurelinuxagent/common/dhcp.py
@@ -49,6 +49,7 @@ class DhcpHandler(object):
         self.gateway = None
         self.routes = None
         self._request_broadcast = False
+        self.skip_cache = False
 
     def run(self):
         """
@@ -56,9 +57,11 @@ class DhcpHandler(object):
         Configure default gateway and routes
         Save wire server endpoint if found
         """
-        if not self.wireserver_route_exists and not self.dhcp_lease_exists:
-            self.send_dhcp_req()
-            self.conf_routes()
+        if self.wireserver_route_exists or self.dhcp_cache_exists:
+            return
+
+        self.send_dhcp_req()
+        self.conf_routes()
 
     def wait_for_network(self):
         """
@@ -104,17 +107,23 @@ class DhcpHandler(object):
         return route_exists
 
     @property
-    def dhcp_lease_exists(self):
+    def dhcp_cache_exists(self):
         """
         Check whether the dhcp options cache exists and contains the
-        wireserver endpoint.
+        wireserver endpoint, unless skip_cache is True.
         :return: True if the cached endpoint was found in the dhcp lease
         """
+        if self.skip_cache:
+            return False
+
         exists = False
+
+        logger.info("checking for dhcp lease cache")
         cached_endpoint = self.osutil.get_dhcp_lease_endpoint()
         if cached_endpoint is not None:
             self.endpoint = cached_endpoint
             exists = True
+        logger.info("cache exists [{0}]".format(exists))
         return exists
 
     def conf_routes(self):
diff --git a/azurelinuxagent/common/osutil/default.py b/azurelinuxagent/common/osutil/default.py
index 2cf34ce4..76d48ca9 100644
--- a/azurelinuxagent/common/osutil/default.py
+++ b/azurelinuxagent/common/osutil/default.py
@@ -73,7 +73,7 @@ class DefaultOSUtil(object):
         userentry = self.get_userentry(username)
         uidmin = None
         try:
-            uidmin_def = fileutil.get_line_startingwith("UID_MIN", 
+            uidmin_def = fileutil.get_line_startingwith("UID_MIN",
                                                         "/etc/login.defs")
             if uidmin_def is not None:
                 uidmin = int(uidmin_def.split()[1])
@@ -215,7 +215,7 @@ class DefaultOSUtil(object):
             pub_path = os.path.join(lib_dir, thumbprint + '.pub')
             pub = crytputil.get_pubkey_from_crt(crt_path)
             fileutil.write_file(pub_path, pub)
-            self.set_selinux_context(pub_path, 
+            self.set_selinux_context(pub_path,
                                      'unconfined_u:object_r:ssh_home_t:s0')
             self.openssl_to_openssh(pub_path, path)
             fileutil.chmod(pub_path, 0o600)
@@ -320,7 +320,7 @@ class DefaultOSUtil(object):
         retcode = self.umount(mount_point, chk_err=chk_err)
         if chk_err and retcode != 0:
             raise OSUtilError("Failed to umount dvd.")
-    
+
     def eject_dvd(self, chk_err=True):
         dvd = self.get_dvd_device()
         retcode = shellutil.run("eject {0}".format(dvd))
@@ -547,6 +547,7 @@ class DefaultOSUtil(object):
         FOOTER_LEASE = "}"
         FORMAT_DATETIME = "%Y/%m/%d %H:%M:%S"
 
+        logger.info("looking for leases in path [{0}]".format(pathglob))
         for lease_file in glob.glob(pathglob):
             leases = open(lease_file).read()
             if HEADER_OPTION in leases:
@@ -557,6 +558,7 @@ class DefaultOSUtil(object):
                     if line.startswith(HEADER_LEASE):
                         cached_endpoint = None
                         has_option_245 = False
+                        expired = True
                     elif HEADER_DNS in line:
                         cached_endpoint = line.replace(HEADER_DNS, '').strip(" ;")
                     elif HEADER_OPTION in line:
@@ -573,10 +575,17 @@ class DefaultOSUtil(object):
                             except:
                                 logger.error("could not parse expiry token '{0}'".format(line))
                     elif FOOTER_LEASE in line:
+                        logger.info("dhcp entry:{0}, 245:{1}, expired:{2}".format(
+                            cached_endpoint, has_option_245, expired))
                         if not expired and cached_endpoint is not None and has_option_245:
                             endpoint = cached_endpoint
-                            # return the first valid entry
-                            break
+                            logger.info("found endpoint [{0}]".format(endpoint))
+                            # we want to return the last valid entry, so
+                            # keep searching
+        if endpoint is not None:
+            logger.info("cached endpoint found [{0}]".format(endpoint))
+        else:
+            logger.info("cached endpoint not found")
         return endpoint
 
     def is_missing_default_route(self):
@@ -761,7 +770,7 @@ class DefaultOSUtil(object):
             return int(ret[1])
         else:
             raise OSUtilError("Failed to get processor cores")
-    
+
     def set_admin_access_to_ip(self, dest_ip):
         #This allows root to access dest_ip
         rm_old= "iptables -D OUTPUT -d {0} -j ACCEPT -m owner --uid-owner 0"
diff --git a/azurelinuxagent/common/protocol/util.py b/azurelinuxagent/common/protocol/util.py
index 98adeeda..7e7a74f1 100644
--- a/azurelinuxagent/common/protocol/util.py
+++ b/azurelinuxagent/common/protocol/util.py
@@ -139,6 +139,7 @@ class ProtocolUtil(object):
         except ProtocolError as e:
             logger.info("WireServer is not responding. Reset endpoint")
             self.dhcp_handler.endpoint = None
+            self.dhcp_handler.skip_cache = True
             raise e
 
     def _detect_metadata_protocol(self):
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index a482e319..a488647e 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -867,7 +867,6 @@ class WireClient(object):
             "x-ms-guest-agent-public-x509-cert": cert
         }
 
-
 class VersionInfo(object):
     def __init__(self, xml_text):
         """
@@ -981,7 +980,6 @@ class SharedConfig(object):
         # Not used currently
         return self
 
-
 class Certificates(object):
     """
     Object containing certificates of host and provisioned user.
diff --git a/azurelinuxagent/common/rdma.py b/azurelinuxagent/common/rdma.py
index 32d46adb..2807c63b 100644
--- a/azurelinuxagent/common/rdma.py
+++ b/azurelinuxagent/common/rdma.py
@@ -21,9 +21,15 @@ Handle packages and modules to enable RDMA for IB networking
 
 import os
 import re
+import threading
+import time
 import azurelinuxagent.common.logger as logger
+import azurelinuxagent.common.utils.fileutil as fileutil
 import azurelinuxagent.common.utils.shellutil as shellutil
 
+dapl_config_paths = ['/etc/dat.conf', '/etc/rdma/dat.conf',
+                     '/usr/local/etc/dat.conf']
+
 
 class RDMAHandler(object):
 
@@ -97,3 +103,134 @@ class RDMAHandler(object):
         ret = shellutil.run('shutdown -r now')
         if ret != 0:
             logger.error('RDMA: Failed to reboot the system')
+
+
+dapl_config_paths = [
+    '/etc/dat.conf', '/etc/rdma/dat.conf', '/usr/local/etc/dat.conf']
+
+class RDMADeviceHandler(object):
+
+    """
+    Responsible for writing RDMA IP and MAC address to the /dev/hvnd_rdma
+    interface.
+    """
+
+    rdma_dev = '/dev/hvnd_rdma'
+    device_check_timeout_sec = 120
+    device_check_interval_sec = 1
+
+    ipv4_addr = None
+    mac_adr = None
+
+    def __init__(self, ipv4_addr, mac_addr):
+        self.ipv4_addr = ipv4_addr
+        self.mac_addr = mac_addr
+
+    def start(self):
+        """
+        Start a thread in the background to process the RDMA tasks and returns.
+        """
+        logger.info("RDMA: starting device processing in the background.")
+        threading.Thread(target=self.process).start()
+
+    def process(self):
+        RDMADeviceHandler.wait_rdma_device(
+            self.rdma_dev, self.device_check_timeout_sec, self.device_check_interval_sec)
+        RDMADeviceHandler.update_dat_conf(dapl_config_paths, self.ipv4_addr)
+        RDMADeviceHandler.write_rdma_config_to_device(
+            self.rdma_dev, self.ipv4_addr, self.mac_addr)
+        RDMADeviceHandler.update_network_interface(self.mac_addr, self.ipv4_addr)
+
+    @staticmethod
+    def update_dat_conf(paths, ipv4_addr):
+        """
+        Looks at paths for dat.conf file and updates the ip address for the
+        infiniband interface.
+        """
+        logger.info("Updating DAPL configuration file")
+        for f in paths:
+            logger.info("RDMA: trying {0}".format(f))
+            if not os.path.isfile(f):
+                logger.info(
+                    "RDMA: DAPL config not found at {0}".format(f))
+                continue
+            logger.info("RDMA: DAPL config is at: {0}".format(f))
+            cfg = fileutil.read_file(f)
+            new_cfg = RDMADeviceHandler.replace_dat_conf_contents(
+                cfg, ipv4_addr)
+            fileutil.write_file(f, new_cfg)
+            logger.info("RDMA: DAPL configuration is updated")
+            return
+
+        raise Exception("RDMA: DAPL configuration file not found at predefined paths")
+
+    @staticmethod
+    def replace_dat_conf_contents(cfg, ipv4_addr):
+        old = "ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 \"\S+ 0\""
+        new = "ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 \"{0} 0\"".format(
+            ipv4_addr)
+        return re.sub(old, new, cfg)
+
+    @staticmethod
+    def write_rdma_config_to_device(path, ipv4_addr, mac_addr):
+        data = RDMADeviceHandler.generate_rdma_config(ipv4_addr, mac_addr)
+        logger.info(
+            "RDMA: Updating device with configuration: {0}".format(data))
+        with open(path, "w") as f:
+            f.write(data)
+        logger.info("RDMA: Updated device with IPv4/MAC addr successfully")
+
+    @staticmethod
+    def generate_rdma_config(ipv4_addr, mac_addr):
+        return 'rdmaMacAddress="{0}" rdmaIPv4Address="{1}"'.format(mac_addr, ipv4_addr)
+
+    @staticmethod
+    def wait_rdma_device(path, timeout_sec, check_interval_sec):
+        logger.info("RDMA: waiting for device={0} timeout={1}s".format(path, timeout_sec))
+        total_retries = timeout_sec/check_interval_sec
+        n = 0
+        while n < total_retries:
+            if os.path.exists(path):
+                logger.info("RDMA: device ready")
+                return
+            logger.verbose(
+                "RDMA: device not ready, sleep {0}s".format(check_interval_sec))
+            time.sleep(check_interval_sec)
+            n += 1
+        logger.error("RDMA device wait timed out")
+        raise Exception("The device did not show up in {0} seconds ({1} retries)".format(
+            timeout_sec, total_retries))
+
+    @staticmethod
+    def update_network_interface(mac_addr, ipv4_addr):
+        netmask=16
+        
+        logger.info("RDMA: will update the network interface with IPv4/MAC")
+
+        if_name=RDMADeviceHandler.get_interface_by_mac(mac_addr)
+        logger.info("RDMA: network interface found: {0}", if_name)
+        logger.info("RDMA: bringing network interface up")
+        if shellutil.run("ifconfig {0} up".format(if_name)) != 0:
+            raise Exception("Could not bring up RMDA interface: {0}".format(if_name))
+
+        logger.info("RDMA: configuring IPv4 addr and netmask on interface")
+        addr = '{0}/{1}'.format(ipv4_addr, netmask)
+        if shellutil.run("ifconfig {0} {1}".format(if_name, addr)) != 0:
+            raise Exception("Could set addr to {1} on {0}".format(if_name, addr))
+        logger.info("RDMA: network address and netmask configured on interface")
+
+    @staticmethod
+    def get_interface_by_mac(mac):
+        ret, output = shellutil.run_get_output("ifconfig -a")
+        if ret != 0:
+            raise Exception("Failed to list network interfaces")
+        output = output.replace('\n', '')
+        match = re.search(r"(eth\d).*(HWaddr|ether) {0}".format(mac), 
+                          output, re.IGNORECASE)
+        if match is None:
+            raise Exception("Failed to get ifname with mac: {0}".format(mac))
+        output = match.group(0)
+        eths = re.findall(r"eth\d", output)
+        if eths is None or len(eths) == 0:
+            raise Exception("ifname with mac: {0} not found".format(mac))
+        return eths[-1]
diff --git a/azurelinuxagent/daemon/main.py b/azurelinuxagent/daemon/main.py
index a8e466bc..431c68a2 100644
--- a/azurelinuxagent/daemon/main.py
+++ b/azurelinuxagent/daemon/main.py
@@ -30,8 +30,10 @@ from azurelinuxagent.common.version import AGENT_LONG_NAME, AGENT_VERSION, \
                                      DISTRO_NAME, DISTRO_VERSION, \
                                      DISTRO_FULL_NAME, PY_VERSION_MAJOR, \
                                      PY_VERSION_MINOR, PY_VERSION_MICRO
+from azurelinuxagent.common.protocol.wire import SHARED_CONF_FILE_NAME
 import azurelinuxagent.common.event as event
 import azurelinuxagent.common.utils.fileutil as fileutil
+from azurelinuxagent.common.utils.textutil import parse_doc, find, getattrib
 from azurelinuxagent.common.osutil import get_osutil
 from azurelinuxagent.common.protocol import get_protocol_util
 from azurelinuxagent.daemon.scvmm import get_scvmm_handler
@@ -41,6 +43,7 @@ from azurelinuxagent.daemon.env import get_env_handler
 from azurelinuxagent.pa.provision import get_provision_handler
 from azurelinuxagent.pa.rdma import get_rdma_handler
 from azurelinuxagent.ga.update import get_update_handler
+from azurelinuxagent.common.rdma import RDMADeviceHandler
 
 def get_daemon_handler():
     return DaemonHandler()
@@ -118,6 +121,52 @@ class DaemonHandler(object):
         self.monitor_handler.run()
 
         self.env_handler.run()
+
+        # Enable RDMA, continue in errors
+        if conf.enable_rdma():
+                logger.info("RDMA capabilities are enabled in configuration")
+                try:
+                    self.setup_rdma_device()
+                except Exception as e:
+                    logger.error("Error setting up rdma device: %s" % e)
+        else:
+            logger.info("RDMA capabilities are not enabled, skipping")
         
         while self.running:
             self.update_handler.run_latest()
+
+
+    def setup_rdma_device(self):
+        logger.verbose("Parsing SharedConfig XML contents for RDMA details")
+        xml_doc = parse_doc(
+            fileutil.read_file(os.path.join(conf.get_lib_dir(), SHARED_CONF_FILE_NAME)))
+        if xml_doc is None:
+            logger.error("Could not parse SharedConfig XML document")
+            return
+        instance_elem = find(xml_doc, "Instance")
+        if not instance_elem:
+            logger.error("Could not find <Instance> in SharedConfig document")
+            return
+
+        rdma_ipv4_addr = getattrib(instance_elem, "rdmaIPv4Address")
+        if not rdma_ipv4_addr:
+            logger.error(
+                "Could not find rdmaIPv4Address attribute on Instance element of SharedConfig.xml document")
+            return
+
+        rdma_mac_addr = getattrib(instance_elem, "rdmaMacAddress")
+        if not rdma_mac_addr:
+            logger.error(
+                "Could not find rdmaMacAddress attribute on Instance element of SharedConfig.xml document")
+            return
+
+        # add colons to the MAC address (e.g. 00155D33FF1D ->
+        # 00:15:5D:33:FF:1D)
+        rdma_mac_addr = ':'.join([rdma_mac_addr[i:i+2]
+                                  for i in range(0, len(rdma_mac_addr), 2)])
+        logger.info("Found RDMA details. IPv4={0} MAC={1}".format(
+            rdma_ipv4_addr, rdma_mac_addr))
+
+        # Set up the RDMA device with collected informatino
+        RDMADeviceHandler(rdma_ipv4_addr, rdma_mac_addr).start()
+        logger.info("RDMA: device is set up")


Test patch:
diff --git a/tests/common/dhcp/test_dhcp.py b/tests/common/dhcp/test_dhcp.py
index 5e4d6085..c27ffe88 100644
--- a/tests/common/dhcp/test_dhcp.py
+++ b/tests/common/dhcp/test_dhcp.py
@@ -65,12 +65,43 @@ class TestDHCP(AgentTestCase):
         self.assertTrue(dhcp_handler.routes is None)
         self.assertTrue(dhcp_handler.gateway is None)
 
-    def test_dhcp_lease_exists(self):
+    def test_dhcp_cache_exists(self):
         dhcp_handler = dhcp.get_dhcp_handler()
         dhcp_handler.osutil = osutil.DefaultOSUtil()
-        with patch.object(osutil.DefaultOSUtil, 'get_dhcp_lease_endpoint', return_value=None):
-            self.assertFalse(dhcp_handler.dhcp_lease_exists)
+        with patch.object(osutil.DefaultOSUtil, 'get_dhcp_lease_endpoint',
+                          return_value=None):
+            self.assertFalse(dhcp_handler.dhcp_cache_exists)
             self.assertEqual(dhcp_handler.endpoint, None)
-        with patch.object(osutil.DefaultOSUtil, 'get_dhcp_lease_endpoint', return_value="foo"):
-            self.assertTrue(dhcp_handler.dhcp_lease_exists)
+        with patch.object(osutil.DefaultOSUtil, 'get_dhcp_lease_endpoint',
+                          return_value="foo"):
+            self.assertTrue(dhcp_handler.dhcp_cache_exists)
             self.assertEqual(dhcp_handler.endpoint, "foo")
+
+    def test_dhcp_skip_cache(self):
+        handler = dhcp.get_dhcp_handler()
+        handler.osutil = osutil.DefaultOSUtil()
+        with patch('os.path.exists', return_value=False):
+            with patch.object(osutil.DefaultOSUtil, 'get_dhcp_lease_endpoint')\
+                    as patch_dhcp_cache:
+                with patch.object(dhcp.DhcpHandler, 'send_dhcp_req') \
+                        as patch_dhcp_send:
+
+                    endpoint = 'foo'
+                    patch_dhcp_cache.return_value = endpoint
+
+                    # endpoint comes from cache
+                    self.assertFalse(handler.skip_cache)
+                    handler.run()
+                    self.assertTrue(patch_dhcp_cache.call_count == 1)
+                    self.assertTrue(patch_dhcp_send.call_count == 0)
+                    self.assertTrue(handler.endpoint == endpoint)
+
+                    # reset
+                    handler.skip_cache = True
+                    handler.endpoint = None
+
+                    # endpoint comes from dhcp request
+                    self.assertTrue(handler.skip_cache)
+                    handler.run()
+                    self.assertTrue(patch_dhcp_cache.call_count == 1)
+                    self.assertTrue(patch_dhcp_send.call_count == 1)
diff --git a/tests/common/osutil/test_default.py b/tests/common/osutil/test_default.py
index 23c5f57e..22813484 100644
--- a/tests/common/osutil/test_default.py
+++ b/tests/common/osutil/test_default.py
@@ -124,5 +124,12 @@ class TestOSUtil(AgentTestCase):
                 self.assertTrue(endpoint is not None)
                 self.assertEqual(endpoint, "168.63.129.16")
 
+    def test_dhcp_lease_multi(self):
+        with patch.object(glob, "glob", return_value=['/var/lib/dhcp/dhclient.eth0.leases']):
+            with patch(open_patch(), mock.mock_open(read_data=load_data("dhcp.leases.multi"))):
+                endpoint = get_osutil(distro_name='ubuntu', distro_version='12.04').get_dhcp_lease_endpoint()
+                self.assertTrue(endpoint is not None)
+                self.assertEqual(endpoint, "second")
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/tests/data/dhcp.leases b/tests/data/dhcp.leases
index 875b13be..cb4f3961 100644
--- a/tests/data/dhcp.leases
+++ b/tests/data/dhcp.leases
@@ -7,14 +7,15 @@ lease {
   option routers 10.0.1.1;
   option dhcp-message-type 5;
   option dhcp-server-identifier 168.63.129.16;
-  option domain-name-servers expired;
+  option domain-name-servers invalid;
   option dhcp-renewal-time 4294967295;
+  option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
   option unknown-245 a8:3f:81:10;
   option dhcp-rebinding-time 4294967295;
   option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
-  renew 4 2015/06/16 16:58:54;
-  rebind 4 2015/06/16 16:58:54;
-  expire 4 2015/06/16 16:58:54;
+  renew 0 2152/07/23 23:27:10;
+  rebind 0 2152/07/23 23:27:10;
+  expire 0 never;
 }
 lease {
   interface "eth0";
@@ -25,15 +26,14 @@ lease {
   option routers 10.0.1.1;
   option dhcp-message-type 5;
   option dhcp-server-identifier 168.63.129.16;
-  option domain-name-servers 168.63.129.16;
+  option domain-name-servers expired;
   option dhcp-renewal-time 4294967295;
-  option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
   option unknown-245 a8:3f:81:10;
   option dhcp-rebinding-time 4294967295;
   option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
-  renew 0 2152/07/23 23:27:10;
-  rebind 0 2152/07/23 23:27:10;
-  expire 0 2152/07/23 23:27:10;
+  renew 4 2015/06/16 16:58:54;
+  rebind 4 2015/06/16 16:58:54;
+  expire 4 2015/06/16 16:58:54;
 }
 lease {
   interface "eth0";
@@ -44,7 +44,7 @@ lease {
   option routers 10.0.1.1;
   option dhcp-message-type 5;
   option dhcp-server-identifier 168.63.129.16;
-  option domain-name-servers invalid;
+  option domain-name-servers 168.63.129.16;
   option dhcp-renewal-time 4294967295;
   option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
   option unknown-245 a8:3f:81:10;
@@ -52,5 +52,5 @@ lease {
   option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
   renew 0 2152/07/23 23:27:10;
   rebind 0 2152/07/23 23:27:10;
-  expire 0 never;
+  expire 0 2152/07/23 23:27:10;
 }
diff --git a/tests/data/dhcp.leases.multi b/tests/data/dhcp.leases.multi
new file mode 100644
index 00000000..cfe9c67d
--- /dev/null
+++ b/tests/data/dhcp.leases.multi
@@ -0,0 +1,57 @@
+lease {
+  interface "eth0";
+  fixed-address 10.0.1.4;
+  server-name "RDE41D2D9BB18C";
+  option subnet-mask 255.255.255.0;
+  option dhcp-lease-time 4294967295;
+  option routers 10.0.1.1;
+  option dhcp-message-type 5;
+  option dhcp-server-identifier 168.63.129.16;
+  option domain-name-servers first;
+  option dhcp-renewal-time 4294967295;
+  option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
+  option unknown-245 a8:3f:81:10;
+  option dhcp-rebinding-time 4294967295;
+  option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
+  renew 0 2152/07/23 23:27:10;
+  rebind 0 2152/07/23 23:27:10;
+  expire 0 2152/07/23 23:27:10;
+}
+lease {
+  interface "eth0";
+  fixed-address 10.0.1.4;
+  server-name "RDE41D2D9BB18C";
+  option subnet-mask 255.255.255.0;
+  option dhcp-lease-time 4294967295;
+  option routers 10.0.1.1;
+  option dhcp-message-type 5;
+  option dhcp-server-identifier 168.63.129.16;
+  option domain-name-servers second;
+  option dhcp-renewal-time 4294967295;
+  option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
+  option unknown-245 a8:3f:81:10;
+  option dhcp-rebinding-time 4294967295;
+  option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
+  renew 0 2152/07/23 23:27:10;
+  rebind 0 2152/07/23 23:27:10;
+  expire 0 2152/07/23 23:27:10;
+}
+lease {
+  interface "eth0";
+  fixed-address 10.0.1.4;
+  server-name "RDE41D2D9BB18C";
+  option subnet-mask 255.255.255.0;
+  option dhcp-lease-time 4294967295;
+  option routers 10.0.1.1;
+  option dhcp-message-type 5;
+  option dhcp-server-identifier 168.63.129.16;
+  option domain-name-servers expired;
+  option dhcp-renewal-time 4294967295;
+  option rfc3442-classless-static-routes 0,10,0,1,1,32,168,63,129,16,10,0,1,1;
+  option unknown-245 a8:3f:81:10;
+  option dhcp-rebinding-time 4294967295;
+  option domain-name "qylsde3bnlhu5dstzf3bav5inc.fx.internal.cloudapp.net";
+  renew 0 2152/07/23 23:27:10;
+  rebind 0 2152/07/23 23:27:10;
+  expire 0 2012/07/23 23:27:10;
+}