Repository: Azure/WALinuxAgent (python)

Issue:
[2.1.5] AutoUpdate WALA error on RHEL-6.8
Description of problem:
waagent fails to run the AutoUpdated WALA process.

Version-Release number of selected component (if applicable):
WALinuxAgent-2.1.5

RHEL Version: RHEL-6.8
Python Version: 2.6.6

How reproducible:
100%

Steps to Reproduce:
1. Prepare a RHEL7.3 VM on Azure East US location(ASM mode) with WALA-2.1.5 installed. Enable wala auto-update in the /etc/waagent.conf:
\# AutoUpdate.Enabled=y
\# AutoUpdate.GAFamily=Prod
Then restart waagent service
\# service waagent restart
2. Wait until the new WALA package is downloaded. Check waagent.log. 

Actual results:
waagent can't run the new WALA version.
There's error logs in waagent.log:

```
2016/08/25 20:20:17.474136 INFO Instantiating Agent WALinuxAgent-2.1.6 from package
2016/08/25 20:20:17.476843 INFO Agent WALinuxAgent-2.1.6 error state: Last Failure: 0.0, Total Failures: 0, Fatal: False
2016/08/25 20:20:17.476980 INFO Ensuring Agent WALinuxAgent-2.1.6 is downloaded
2016/08/25 20:20:17.738310 INFO Agent WALinuxAgent-2.1.6 downloaded from https://rdfepirv2bl2prdstr01.blob.core.windows.net/7d89d439b79f4452950452399add2c90/Microsoft.OSTCLinuxAgent__Prod__2.1.6
2016/08/25 20:20:17.799473 INFO Agent WALinuxAgent-2.1.6 unpacked successfully to /var/lib/waagent/WALinuxAgent-2.1.6
2016/08/25 20:20:17.877585 INFO Agent WALinuxAgent-2.1.6 loaded manifest from /var/lib/waagent/WALinuxAgent-2.1.6/HandlerManifest.json
2016/08/25 20:20:17.962055 INFO Agent WALinuxAgent-2.1.6 error state: Last Failure: 0.0, Total Failures: 0, Fatal: False
2016/08/25 20:20:17.977174 INFO Agent WALinuxAgent-2.1.6 downloaded successfully
2016/08/25 20:20:18.021588 INFO Event: name=WALinuxAgent, op=Install, message=Agent WALinuxAgent-2.1.6 downloaded successfully
2016/08/25 20:20:18.102944 INFO Agent WALinuxAgent-2.1.5 discovered WALinuxAgent-2.1.6 as an update and will exit
2016/08/25 20:21:12.959822 INFO Agent WALinuxAgent-2.1.5 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' is successfully running
2016/08/25 20:21:13.048011 INFO Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.1.5 launched with command 'python -u /usr/sbin/waagent -run-exthandlers' is successfully running
2016/08/25 20:21:13.146289 INFO Instantiating Agent WALinuxAgent-2.1.6 from disk
2016/08/25 20:21:13.195095 INFO Agent WALinuxAgent-2.1.6 error state: Last Failure: 0.0, Total Failures: 0, Fatal: False
2016/08/25 20:21:13.230193 INFO Ensuring Agent WALinuxAgent-2.1.6 is downloaded
2016/08/25 20:21:13.235666 INFO Agent WALinuxAgent-2.1.6 was previously downloaded - skipping download
2016/08/25 20:21:13.242621 INFO Agent WALinuxAgent-2.1.6 loaded manifest from /var/lib/waagent/WALinuxAgent-2.1.6/HandlerManifest.json
2016/08/25 20:21:13.328814 INFO Determined Agent WALinuxAgent-2.1.6 to be the latest agent
2016/08/25 20:21:13.343555 WARNING Agent WALinuxAgent-2.1.6 launched with command 'python -u bin/WALinuxAgent-2.1.6-py2.7.egg -run-exthandlers' failed with exception: execv() argument 1 must be encoded string without NULL bytes, not str
2016/08/25 20:21:13.566290 ERROR Event: name=WALinuxAgent, op=Enable, message=Agent WALinuxAgent-2.1.6 launched with command 'python -u bin/WALinuxAgent-2.1.6-py2.7.egg -run-exthandlers' failed with exception: execv() argument 1 must be encoded string without NULL bytes, not str
2016/08/25 20:21:13.675956 WARNING Agent WALinuxAgent-2.1.6 is permanently blacklisted
2016/08/25 20:21:13.749416 INFO Installed Agent WALinuxAgent-2.1.5 is the most current agent
2016/08/25 20:21:13.756486 INFO Agent WALinuxAgent-2.1.5 launched with command 'python -u /usr/sbin/waagent -run-exthandlers'

\# cat /var/lib/waagent/WALinuxAgent-2.1.6/error.json
{"was_fatal": true, "failure_count": 1, "last_failure": 1472127673.6758201}
```

Additional info:
1. Only exists on RHEL-6. Doesn't exist on RHEL-7.
2. If install WALA-2.1.6 directly, there's no error log. Only exists during AutoUpdate.

Pull request description:
ensure null bytes are not present in shlex input
Python 2.6 contains a bug in `shlex_split` which can be avoided by encoding the input beforehand. This does not work for Python 3 however, so we need to have a selective function which only encodes when necessary. 

- adds `textutil.safe_shlex_split`
- fixes #389 

/cc @brendandixon 

Patch:
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index 287da33c..29a1663f 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -269,7 +269,7 @@ def ext_status_to_v1(ext_name, ext_status):
         "timestampUTC": timestamp
     }
     if len(v1_sub_status) != 0:
-        v1_ext_status['status']['substatus'] = v1_sub_status
+        v1_ext_status['substatus'] = v1_sub_status
     return v1_ext_status
 
 
diff --git a/azurelinuxagent/common/utils/textutil.py b/azurelinuxagent/common/utils/textutil.py
index f03c7e67..6d460cb1 100644
--- a/azurelinuxagent/common/utils/textutil.py
+++ b/azurelinuxagent/common/utils/textutil.py
@@ -277,3 +277,11 @@ def b64encode(s):
     if PY_VERSION_MAJOR > 2:
         return base64.b64encode(bytes(s, 'utf-8')).decode('utf-8')
     return base64.b64encode(s)
+
+
+def safe_shlex_split(s):
+    import shlex
+    from azurelinuxagent.common.version import PY_VERSION
+    if PY_VERSION[:2] == (2, 6):
+        return shlex.split(s.encode('utf-8'))
+    return shlex.split(s)
diff --git a/azurelinuxagent/daemon/resourcedisk/default.py b/azurelinuxagent/daemon/resourcedisk/default.py
index 9ffe2809..a5f4cf66 100644
--- a/azurelinuxagent/daemon/resourcedisk/default.py
+++ b/azurelinuxagent/daemon/resourcedisk/default.py
@@ -106,10 +106,7 @@ class ResourceDiskHandler(object):
             raise ResourceDiskError("Could not determine partition info for "
                                     "{0}: {1}".format(device, ret[1]))
 
-        force_option = 'F'
-        if fs == 'xfs':
-            force_option = 'f'
-        mkfs_string = "mkfs.{0} {1} -{2}".format(fs, partition, force_option)
+        mkfs_string = "mkfs.{0} {1} -F".format(fs, partition)
 
         if "gpt" in ret[1]:
             logger.info("GPT detected, finding partitions")
diff --git a/azurelinuxagent/ga/update.py b/azurelinuxagent/ga/update.py
index 37774d29..5eff7e53 100644
--- a/azurelinuxagent/ga/update.py
+++ b/azurelinuxagent/ga/update.py
@@ -16,12 +16,12 @@
 #
 # Requires Python 2.4+ and Openssl 1.0+
 #
+
 import glob
 import json
 import os
 import platform
 import re
-import shlex
 import shutil
 import signal
 import subprocess
@@ -130,7 +130,7 @@ class UpdateHandler(object):
         try:
 
             # Launch the correct Python version for python-based agents
-            cmds = shlex.split(agent_cmd)
+            cmds = textutil.safe_shlex_split(agent_cmd)
             if cmds[0].lower() == "python":
                 cmds[0] = get_python_cmd()
                 agent_cmd = " ".join(cmds)
diff --git a/setup.py b/setup.py
index 7303b62c..887a557e 100755
--- a/setup.py
+++ b/setup.py
@@ -147,7 +147,7 @@ class install(_install):
         ('lnx-distro=', None, 'target Linux distribution'),
         ('lnx-distro-version=', None, 'target Linux distribution version'),
         ('lnx-distro-fullname=', None, 'target Linux distribution full name'),
-        ('register-service', None, 'register as startup service and start'),
+        ('register-service', None, 'register as startup service'),
         ('skip-data-files', None, 'skip data files installation'),
     ]
 
@@ -172,16 +172,14 @@ class install(_install):
     def run(self):
         _install.run(self)
         if self.register_service:
-            osutil = get_osutil()
-            osutil.register_agent_service()
-            osutil.start_agent_service()
+            get_osutil().register_agent_service()
 
 
 setuptools.setup(
     name=AGENT_NAME,
     version=AGENT_VERSION,
     long_description=AGENT_DESCRIPTION,
-    author='Microsoft Corporation',
+    author='Yue Zhang, Stephen Zarkos, Eric Gable',
     author_email='walinuxagent@microsoft.com',
     platforms='Linux',
     url='https://github.com/Azure/WALinuxAgent',


Test patch:
diff --git a/tests/ga/test_update.py b/tests/ga/test_update.py
index f4b33207..0e07119d 100644
--- a/tests/ga/test_update.py
+++ b/tests/ga/test_update.py
@@ -438,7 +438,8 @@ class TestGuestAgent(UpdateTestCase):
         agent = GuestAgent(path=self.agent_path)
         agent._unpack()
         agent._load_manifest()
-        self.assertEqual(agent.manifest.get_enable_command(), agent.get_agent_cmd())
+        self.assertEqual(agent.manifest.get_enable_command(),
+                         agent.get_agent_cmd())
         return
 
     @patch("azurelinuxagent.ga.update.GuestAgent._ensure_downloaded")
@@ -992,13 +993,14 @@ class TestUpdate(UpdateTestCase):
 
         agent = self.update_handler.get_latest_agent()
         args, kwargs = self._test_run_latest()
-        cmds = shlex.split(agent.get_agent_cmd())
+        cmds = textutil.safe_shlex_split(agent.get_agent_cmd())
         if cmds[0].lower() == "python":
             cmds[0] = get_python_cmd()
 
         self.assertEqual(args[0], cmds)
         self.assertEqual(True, 'cwd' in kwargs)
         self.assertEqual(agent.get_agent_dir(), kwargs['cwd'])
+        self.assertEqual(False, '\x00' in cmds[0])
         return
 
     def test_run_latest_polls_and_waits_for_success(self):