Repository: google/mobly

Issue:
Unable to launch MBS on xiaomi devices
Could it be that the recent changes in shell has broken the fragile regex matching of adb output...
This happened on mac:
```
$ snippet_shell.py com.google.android.mobly.snippet.bundled
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/snippet_shell.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/angli/Developer/mobly/tools/snippet_shell.py", line 64, in <module>
    SnippetShell(args.package[0]).main(args.serial)
  File "/Users/angli/Developer/mobly/mobly/controllers/android_device_lib/jsonrpc_shell_base.py", line 91, in main
    self.start_console()
  File "/Users/angli/Developer/mobly/mobly/controllers/android_device_lib/jsonrpc_shell_base.py", line 76, in start_console
    self._start_services(console_env)
  File "/Users/angli/Developer/mobly/tools/snippet_shell.py", line 41, in _start_services
    self._ad.load_snippet(name='snippet', package=self._package)
  File "/Users/angli/Developer/mobly/mobly/controllers/android_device.py", line 629, in load_snippet
    self._start_jsonrpc_client(client)
  File "/Users/angli/Developer/mobly/mobly/controllers/android_device.py", line 660, in _start_jsonrpc_client
    client.check_app_installed()
  File "/Users/angli/Developer/mobly/mobly/controllers/android_device_lib/snippet_client.py", line 110, in check_app_installed
    target_name = match.group(3)
AttributeError: 'NoneType' object has no attribute 'group'
```


Discussion:
Launching snippets still works for me, so I think I'd need to see the test_run_details.txt on this one.
Im using snippet shell. I think this might be Mac specific.

On Mar 30, 2017 8:45 PM, "Alexander Dorokhine" <notifications@github.com>
wrote:

> Launching snippets still works for me, so I think I'd need to see the
> test_run_details.txt on this one.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/google/mobly/issues/174#issuecomment-290608278>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ACI25RIHI4XRSB1wnb6P3n_awMwGz-oWks5rrHbdgaJpZM4MvCmK>
> .
>

Can you turn up the logging to DEBUG then? I think I'd need to see what
commands are going out and what's coming back.

On Thu, Mar 30, 2017 at 8:51 PM, Ang Li <notifications@github.com> wrote:

> Im using snippet shell. I think this might be Mac specific.
>
> On Mar 30, 2017 8:45 PM, "Alexander Dorokhine" <notifications@github.com>
> wrote:
>
> > Launching snippets still works for me, so I think I'd need to see the
> > test_run_details.txt on this one.
> >
> > —
> > You are receiving this because you authored the thread.
> > Reply to this email directly, view it on GitHub
> > <https://github.com/google/mobly/issues/174#issuecomment-290608278>, or
> mute
> > the thread
> > <https://github.com/notifications/unsubscribe-
> auth/ACI25RIHI4XRSB1wnb6P3n_awMwGz-oWks5rrHbdgaJpZM4MvCmK>
> > .
> >
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/google/mobly/issues/174#issuecomment-290608896>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AC3EatMx2vdHqcqY_RK7g1drR1FhlT1sks5rrHgugaJpZM4MvCmK>
> .
>

It works on nexus...
Looks like it's an NBU specific issue...
So from what I can tell, this might be a regression from the changes in shell.
We run the following cmd to find snippet package:
```
ad.adb.shell(r'pm list package | tr -d "\r" | grep ^package:%s$' % 'com.google.android.mobly.snippet.bundled')
```

In the old shell, the `tr` and `grep` were done on the host machine, but now it looks like `tr` and `grep` are executed on the Android device's shell. Some NBU devices don't have `tr`, causing the cmd to fail with `b'/system/bin/sh: tr: not found\r\n'`.

Suggestions?
Ok, we just need to use `shell=True` for grep wrapper

Resolving patch:
diff --git a/mobly/controllers/android_device_lib/jsonrpc_client_base.py b/mobly/controllers/android_device_lib/jsonrpc_client_base.py
--- a/mobly/controllers/android_device_lib/jsonrpc_client_base.py
+++ b/mobly/controllers/android_device_lib/jsonrpc_client_base.py
@@ -40,6 +40,7 @@
 
 import json
 import logging
+import re
 import socket
 import threading
 import time
@@ -231,24 +232,28 @@ def close(self):
             self._conn.close()
             self._conn = None
 
-    def _adb_grep_wrapper(self, adb_shell_cmd):
-        """A wrapper for the specific usage of adb shell grep in this class.
+    def _grep(self, regex, output):
+        """Similar to linux's `grep`, this returns the line in an output stream
+        that matches a given regex pattern.
 
-        This surpresses AdbError if the grep fails to find anything.
+        This function is specifically used to grep strings from AdbProxy's
+        output. We have to do this in Python instead of using cli tools because
+        we need to support windows which does not have `grep` in all vesions.
 
         Args:
-            adb_shell_cmd: string, a grep command to execute on the device.
+            regex: string, a regex that matches the expected pattern.
+            output: byte string, the raw output of the adb cmd.
 
         Returns:
-            The stdout of the grep result if the grep found something, False
-            otherwise.
+            A list of strings, all of which are output lines that matches the
+            regex pattern.
         """
-        try:
-            return self._adb.shell(adb_shell_cmd).decode('utf-8').rstrip()
-        except adb.AdbError as e:
-            if (e.ret_code == 1) and (not e.stdout) and (not e.stderr):
-                return False
-            raise
+        lines = output.decode('utf-8').strip().splitlines()
+        results = []
+        for line in lines:
+            if re.search(regex, line):
+                results.append(line.strip())
+        return results
 
     def _cmd(self, command, uid=None):
         """Send a command to the server.
diff --git a/mobly/controllers/android_device_lib/sl4a_client.py b/mobly/controllers/android_device_lib/sl4a_client.py
--- a/mobly/controllers/android_device_lib/sl4a_client.py
+++ b/mobly/controllers/android_device_lib/sl4a_client.py
@@ -15,15 +15,17 @@
 # limitations under the License.
 """JSON RPC interface to android scripting engine."""
 
+import re
+
 from mobly.controllers.android_device_lib import adb
 from mobly.controllers.android_device_lib import jsonrpc_client_base
 
 DEVICE_SIDE_PORT = 8080
 
 _LAUNCH_CMD = (
-    "am start -a com.googlecode.android_scripting.action.LAUNCH_SERVER "
-    "--ei com.googlecode.android_scripting.extra.USE_SERVICE_PORT {} "
-    "com.googlecode.android_scripting/.activity.ScriptingLayerServiceLauncher")
+    'am start -a com.googlecode.android_scripting.action.LAUNCH_SERVER '
+    '--ei com.googlecode.android_scripting.extra.USE_SERVICE_PORT %s '
+    'com.googlecode.android_scripting/.activity.ScriptingLayerServiceLauncher')
 
 
 class Sl4aClient(jsonrpc_client_base.JsonRpcClientBase):
@@ -40,12 +42,14 @@ def __init__(self, host_port, adb_proxy):
             adb_proxy: (adb.AdbProxy) The adb proxy to use to start the app.
         """
         super(Sl4aClient, self).__init__(
-            host_port=host_port, device_port=DEVICE_SIDE_PORT, app_name='SL4A',
+            host_port=host_port,
+            device_port=DEVICE_SIDE_PORT,
+            app_name='SL4A',
             adb_proxy=adb_proxy)
 
     def _do_start_app(self):
         """Overrides superclass."""
-        self._adb.shell(_LAUNCH_CMD.format(self.device_port))
+        self._adb.shell(_LAUNCH_CMD % self.device_port)
 
     def stop_app(self):
         """Overrides superclass."""
@@ -53,8 +57,8 @@ def stop_app(self):
 
     def check_app_installed(self):
         """Overrides superclass."""
-        if not self._adb_grep_wrapper(
-            "pm list package | grep com.googlecode.android_scripting"):
+        out = self._adb('pm list package')
+        if not self._grep('com.googlecode.android_scripting', out):
             raise jsonrpc_client_base.AppStartError(
-                '%s is not installed on %s' % (
-                self.app_name, self._adb.serial))
+                '%s is not installed on %s' % (self.app_name,
+                                               self._adb.serial))
diff --git a/mobly/controllers/android_device_lib/snippet_client.py b/mobly/controllers/android_device_lib/snippet_client.py
--- a/mobly/controllers/android_device_lib/snippet_client.py
+++ b/mobly/controllers/android_device_lib/snippet_client.py
@@ -23,9 +23,8 @@
 _INSTRUMENTATION_RUNNER_PACKAGE = (
     'com.google.android.mobly.snippet.SnippetRunner')
 
-_LAUNCH_CMD = (
-    'am instrument -w -e action start -e port %s %s/' +
-    _INSTRUMENTATION_RUNNER_PACKAGE)
+_LAUNCH_CMD = ('am instrument -w -e action start -e port %s %s/' +
+               _INSTRUMENTATION_RUNNER_PACKAGE)
 
 _STOP_CMD = (
     'am instrument -w -e action stop %s/' + _INSTRUMENTATION_RUNNER_PACKAGE)
@@ -83,7 +82,7 @@ def stop_app(self):
         # print the failure stack trace if there was any, and reap it from the
         # process table.
         if self._proc:
-          utils.stop_standing_subprocess(self._proc)
+            utils.stop_standing_subprocess(self._proc)
         self.log.debug('Stopping snippet apk %s', self.package)
         out = self._adb.shell(_STOP_CMD % self.package).decode('utf-8')
         if 'OK (0 tests)' not in out:
@@ -93,28 +92,26 @@ def stop_app(self):
     def check_app_installed(self):
         """Overrides superclass."""
         # Check that the Mobly Snippet app is installed.
-        if not self._adb_grep_wrapper(
-                r'pm list package | tr -d "\r" | grep "^package:%s$"' %
-                self.package):
+        out = self._adb.shell('pm list package')
+        if not self._grep('^package:%s$' % self.package, out):
             raise jsonrpc_client_base.AppStartError(
                 '%s is not installed on %s' % (self.package, self._serial))
         # Check that the app is instrumented.
-        out = self._adb_grep_wrapper(
-            r'pm list instrumentation | tr -d "\r" | grep ^instrumentation:%s/%s'
-            % (self.package, _INSTRUMENTATION_RUNNER_PACKAGE))
-        if not out:
+        out = self._adb.shell('pm list instrumentation')
+        matched_out = self._grep('^instrumentation:%s/%s' % (
+            self.package, _INSTRUMENTATION_RUNNER_PACKAGE), out)
+        if not matched_out:
             raise jsonrpc_client_base.AppStartError(
                 '%s is installed on %s, but it is not instrumented.' %
                 (self.package, self._serial))
-        match = re.search(r'^instrumentation:(.*)\/(.*) \(target=(.*)\)$', out)
+        match = re.search('^instrumentation:(.*)\/(.*) \(target=(.*)\)$',
+                          matched_out[0])
         target_name = match.group(3)
         # Check that the instrumentation target is installed if it's not the
         # same as the snippet package.
         if target_name != self.package:
-            out = self._adb_grep_wrapper(
-                r'pm list package | tr -d "\r" | grep ^package:%s$' %
-                target_name)
-            if not out:
+            out = self._adb.shell('pm list package')
+            if not self._grep('^package:%s$' % target_name, out):
                 raise jsonrpc_client_base.AppStartError(
                     'Instrumentation target %s is not installed on %s' %
                     (target_name, self._serial))


Test patch:
diff --git a/tests/mobly/controllers/android_device_lib/snippet_client_test.py b/tests/mobly/controllers/android_device_lib/snippet_client_test.py
--- a/tests/mobly/controllers/android_device_lib/snippet_client_test.py
+++ b/tests/mobly/controllers/android_device_lib/snippet_client_test.py
@@ -30,30 +30,35 @@
 MOCK_MISSING_PACKAGE_NAME = 'not.installed'
 JSONRPC_BASE_PACKAGE = 'mobly.controllers.android_device_lib.jsonrpc_client_base.JsonRpcClientBase'
 
+
 class MockAdbProxy(object):
     def __init__(self, **kwargs):
         self.apk_not_installed = kwargs.get('apk_not_installed', False)
         self.apk_not_instrumented = kwargs.get('apk_not_instrumented', False)
         self.target_not_installed = kwargs.get('target_not_installed', False)
 
-    def shell(self, params):
+    def shell(self, params, shell=False):
         if 'pm list package' in params:
             if self.apk_not_installed:
                 return b''
             if self.target_not_installed and MOCK_MISSING_PACKAGE_NAME in params:
                 return b''
-            return bytes(r'package:%s\r' % MOCK_PACKAGE_NAME, 'utf-8')
+            return bytes('package:%s' % MOCK_PACKAGE_NAME, 'utf-8')
         elif 'pm list instrumentation' in params:
             if self.apk_not_instrumented:
                 return b''
             if self.target_not_installed:
-                return bytes(r'instrumentation:{p}\r/{r} (target={mp})'.format(
+                return bytes(
+                    'instrumentation:{p}/{r} (target={mp})'.format(
+                        p=MOCK_PACKAGE_NAME,
+                        r=snippet_client._INSTRUMENTATION_RUNNER_PACKAGE,
+                        mp=MOCK_MISSING_PACKAGE_NAME),
+                    'utf-8')
+            return bytes(
+                'instrumentation:{p}/{r} (target={p})'.format(
                     p=MOCK_PACKAGE_NAME,
-                    r=snippet_client._INSTRUMENTATION_RUNNER_PACKAGE,
-                    mp=MOCK_MISSING_PACKAGE_NAME), 'utf-8')
-            return bytes(r'instrumentation:{p}\r/{r} (target={p})'.format(
-                p=MOCK_PACKAGE_NAME,
-                r=snippet_client._INSTRUMENTATION_RUNNER_PACKAGE), 'utf-8')
+                    r=snippet_client._INSTRUMENTATION_RUNNER_PACKAGE),
+                'utf-8')
 
     def __getattr__(self, name):
         """All calls to the none-existent functions in adb proxy would