Repository: amoffat/sh
Pull request #519 (2020-04-27 04:35:29)

Release 1.13.0
Going to merge this tomorrow. Any eyes to catch last minute fixes would be welcome.


Patch:
diff --git a/.gitignore b/.gitignore
index 78ec7b1..1deb479 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,4 @@
 __pycache__/
 *.py[co]
+.tox
+.coverage
diff --git a/.travis.yml b/.travis.yml
index 881c82e..e30fcb6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -4,14 +4,16 @@ os:
 language: python
 
 python:
-    - 2.6
+    # - 2.6 No longer supported on Travis
     - 2.7
-    - pypy
-    - 3.3
+    # - 3.3 No longer supported on Travis
     - 3.4
     - 3.5
     - 3.6
-    - pypy-5.3.1
+    - 3.7
+    - 3.8
+    - pypy
+    - pypy3.5
 
 before_script:
     - pip install -r requirements-dev.txt
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3ff3904..bdb8661 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,28 @@
 # Changelog
 
+## 1.13.0 - 4/27/20
+*   minor Travis CI fixes [#492](https://github.com/amoffat/sh/pull/492)
+*   bugfix for boolean long options not respecting `_long_prefix` [#488](https://github.com/amoffat/sh/pull/488)
+*   fix deprecation warning on Python 3.6 regexes [#482](https://github.com/amoffat/sh/pull/482)
+*   `_pass_fds` and `_close_fds` special kwargs for controlling file descriptor inheritance in child.
+*   more efficiently closing inherited fds [#406](https://github.com/amoffat/sh/issues/406)
+*   bugfix where passing invalid dictionary to `_env` will cause a mysterious child 255 exit code. [#497](https://github.com/amoffat/sh/pull/497)
+*   bugfix where `_in` using 0 or `sys.stdin` wasn't behaving like a TTY, if it was in fact a TTY. [#514](https://github.com/amoffat/sh/issues/514)
+*   bugfix where `help(sh)` raised an exception [#455](https://github.com/amoffat/sh/issues/455)
+*   bugfix fixing broken interactive ssh tutorial from docs
+*   change to automatic tty merging into a single pty if `_tty_in=True` and `_tty_out=True`
+*   introducing `_unify_ttys`, default False, which allows explicit tty merging into single pty
+*   contrib command for `ssh` connections requiring passwords
+*   performance fix for polling output too fast when using `_iter` [#462](https://github.com/amoffat/sh/issues/462)
+*   execution contexts can now be used in python shell [#466](https://github.com/amoffat/sh/pull/466)
+*   bugfix `ErrorReturnCode` instances can now be pickled
+*   bugfix passing empty string or `None` for `_in` hanged [#427](https://github.com/amoffat/sh/pull/427)
+*   bugfix where passing a filename or file-like object to `_out` wasn't using os.dup2 [#449](https://github.com/amoffat/sh/issues/449)
+*   regression make `_fg` work with `_cwd` again [#330](https://github.com/amoffat/sh/issues/330)
+*   an invalid `_cwd` now raises a `ForkException` not an `OSError`.
+*   AIX support [#477](https://github.com/amoffat/sh/issues/477)
+*   added a `timeout=None` param to `RunningCommand.wait()` [#515](https://github.com/amoffat/sh/issues/515)
+
 ## 1.12.14 - 6/6/17
 *   bugfix for poor sleep performance [#378](https://github.com/amoffat/sh/issues/378)
 *   allow passing raw integer file descriptors for `_out` and `_err` handlers
diff --git a/README.rst b/README.rst
index 73b68c4..e583178 100644
--- a/README.rst
+++ b/README.rst
@@ -7,6 +7,9 @@
 .. image:: https://img.shields.io/pypi/v/sh.svg?style=flat-square
     :target: https://pypi.python.org/pypi/sh
     :alt: Version
+.. image:: https://img.shields.io/pypi/dm/sh.svg?style=flat-square
+    :target: https://pypi.python.org/pypi/sh
+    :alt: Downloads Status
 .. image:: https://img.shields.io/pypi/pyversions/sh.svg?style=flat-square
     :target: https://pypi.python.org/pypi/sh
     :alt: Python Versions
@@ -19,8 +22,8 @@
 
 |
 
-sh is a full-fledged subprocess replacement for Python 2.6 - 3.6, PyPy and PyPy3
-that allows you to call any program as if it were a function:
+sh is a full-fledged subprocess replacement for Python 2.6 - 3.8, PyPy and PyPy3
+that allows you to call *any* program as if it were a function:
 
 .. code:: python
 
@@ -49,21 +52,24 @@ Developers
 Testing
 -------
 
-First install the development requirements::
+I've included a Docker test suite in the `docker_test_suit/` folder.  To build the image, `cd` into that directory and
+run::
 
-    $> pip install -r requirements-dev.txt
+    $> ./build.sh
 
-The run the tests for all Python versions on your system::
+This will install ubuntu 18.04 LTS and all python versions from 2.6-3.8.  Once it's done, stay in that directory and
+run::
 
-    $> python sh.py test
+    $> ./run.sh
 
-To run a single test for all environments::
+This will mount your local code directory into the container and start the test suite, which will take a long time to
+run.  If you wish to run a single test, you may pass that test to `./run.sh`::
 
-    $> python sh.py test FunctionalTests.test_unicode_arg
+    $> ./run.sh FunctionalTests.test_unicode_arg
 
 To run a single test for a single environment::
 
-    $> python sh.py test -e 3.4 FunctionalTests.test_unicode_arg
+    $> ./run.sh -e 3.4 FunctionalTests.test_unicode_arg
 
 Coverage
 --------
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 5277a92..909e3e4 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -3,4 +3,3 @@ coverage==4.2
 coveralls==1.1
 docopt==0.6.2
 docutils==0.12
-requests==2.12.1
diff --git a/requirements-docs.txt b/requirements-docs.txt
index dbaddca..da36353 100644
--- a/requirements-docs.txt
+++ b/requirements-docs.txt
@@ -2,7 +2,7 @@ alabaster==0.7.9
 Babel==2.3.4
 docutils==0.12
 imagesize==0.7.1
-Jinja2==2.8
+Jinja2==2.10.3
 MarkupSafe==0.23
 Pygments==2.1.3
 pytz==2016.7
diff --git a/setup.py b/setup.py
index 7c3d1c1..97cba90 100644
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@ from setuptools import setup
 HERE = dirname(abspath(__file__))
 
 author = "Andrew Moffat"
-author_email = "andrew.robert.moffat@gmail.com"
+author_email = "arwmoffat@gmail.com"
 keywords = ["subprocess", "process", "shell", "launch", "program"]
 
 
@@ -48,6 +48,8 @@ setup(
         "Programming Language :: Python :: 3.4",
         "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: Implementation :: CPython",
         "Programming Language :: Python :: Implementation :: PyPy",
         "Topic :: Software Development :: Build Tools",
diff --git a/sh.py b/sh.py
index 5b250e1..a3758c1 100644
--- a/sh.py
+++ b/sh.py
@@ -2,7 +2,7 @@
 http://amoffat.github.io/sh/
 """
 #===============================================================================
-# Copyright (C) 2011-2017 by Andrew Moffat
+# Copyright (C) 2011-2020 by Andrew Moffat
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
 # of this software and associated documentation files (the "Software"), to deal
@@ -24,7 +24,7 @@ http://amoffat.github.io/sh/
 #===============================================================================
 
 
-__version__ = "1.12.14"
+__version__ = "1.13.0"
 __project_url__ = "https://github.com/amoffat/sh"
 
 
@@ -50,7 +50,9 @@ from types import ModuleType, GeneratorType
 from functools import partial
 import inspect
 import tempfile
+import warnings
 import stat
+from collections import deque
 import glob as glob_module
 import ast
 from contextlib import contextmanager
@@ -87,7 +89,12 @@ else:
     from io import BytesIO as iocStringIO
     from Queue import Queue, Empty
 
-IS_OSX = platform.system() == "Darwin"
+try:
+    from shlex import quote as shlex_quote # here from 3.3 onward
+except ImportError:
+    from pipes import quote as shlex_quote # undocumented before 2.7
+
+IS_MACOS = platform.system() in ("AIX", "Darwin")
 THIS_DIR = os.path.dirname(os.path.realpath(__file__))
 SH_LOGGER_NAME = __name__
 
@@ -360,10 +367,14 @@ class ErrorReturnCode(Exception):
 
     truncate_cap = 750
 
+    def __reduce__(self):
+        return (self.__class__, (self.full_cmd, self.stdout, self.stderr, self.truncate))
+
     def __init__(self, full_cmd, stdout, stderr, truncate=True):
         self.full_cmd = full_cmd
         self.stdout = stdout
         self.stderr = stderr
+        self.truncate = truncate
 
         exc_stdout = self.stdout
         if truncate:
@@ -387,15 +398,22 @@ class ErrorReturnCode(Exception):
             stderr=exc_stderr.decode(DEFAULT_ENCODING, "replace")
         )
 
+        if not IS_PY3:
+            # Exception messages should be treated as an API which takes native str type on both
+            # Python2 and Python3.  (Meaning, it's a byte string on Python2 and a text string on
+            # Python3)
+            msg = encode_to_py3bytes_or_py2str(msg)
+
         super(ErrorReturnCode, self).__init__(msg)
 
 
 class SignalException(ErrorReturnCode): pass
 class TimeoutException(Exception):
     """ the exception thrown when a command is killed because a specified
-    timeout (via _timeout) was hit """
-    def __init__(self, exit_code):
+    timeout (via _timeout or .wait(timeout)) was hit """
+    def __init__(self, exit_code, full_cmd):
         self.exit_code = exit_code
+        self.full_cmd = full_cmd
         super(Exception, self).__init__()
 
 SIGNALS_THAT_SHOULD_THROW_EXCEPTION = set((
@@ -421,7 +439,7 @@ class CommandNotFound(AttributeError): pass
 
 
 
-rc_exc_regex = re.compile("(ErrorReturnCode|SignalException)_((\d+)|SIG[a-zA-Z]+)")
+rc_exc_regex = re.compile(r"(ErrorReturnCode|SignalException)_((\d+)|SIG[a-zA-Z]+)")
 rc_exc_cache = {}
 
 SIGNAL_MAPPING = {}
@@ -517,6 +535,8 @@ def glob(path, *args, **kwargs):
 glob_module.glob = glob
 
 
+def canonicalize(path):
+    return os.path.abspath(os.path.expanduser(path))
 
 
 def which(program, paths=None):
@@ -536,7 +556,7 @@ def which(program, paths=None):
     # if there's a path component, then we've specified a path to the program,
     # and we should just test if that program is executable.  if it is, return
     if fpath:
-        program = os.path.abspath(os.path.expanduser(program))
+        program = canonicalize(program)
         if is_exe(program):
             found_path = program
 
@@ -552,7 +572,7 @@ def which(program, paths=None):
             paths_to_search.extend(env_paths)
 
         for path in paths_to_search:
-            exe_file = os.path.join(path, program)
+            exe_file = os.path.join(canonicalize(path), program)
             if is_exe(exe_file):
                 found_path = exe_file
                 break
@@ -660,7 +680,7 @@ class RunningCommand(object):
     finishes, RunningCommand is smart enough to translate exit codes to
     exceptions. """
 
-    # these are attributes that we allow to passthrough to OProc for
+    # these are attributes that we allow to passthrough to OProc
     _OProc_attr_whitelist = set((
         "signal",
         "terminate",
@@ -679,8 +699,7 @@ class RunningCommand(object):
 
     def __init__(self, cmd, call_args, stdin, stdout, stderr):
         """
-            cmd is an array, where each element is encoded as bytes (PY3) or str
-            (PY2)
+            cmd is a list, where each element is encoded as bytes (PY3) or str (PY2)
         """
 
         # self.ran is used for auditing what actually ran.  for example, in
@@ -692,13 +711,13 @@ class RunningCommand(object):
         # arguments are the encoding we pass into _encoding, which falls back to
         # the system's encoding
         enc = call_args["encoding"]
-        self.ran = " ".join([arg.decode(enc, "ignore") for arg in cmd])
+        self.ran = " ".join([shlex_quote(arg.decode(enc, "ignore")) for arg in cmd])
 
         self.call_args = call_args
         self.cmd = cmd
 
         self.process = None
-        self._process_completed = False
+        self._waited_until_completion = False
         should_wait = True
         spawn_process = True
 
@@ -753,7 +772,7 @@ class RunningCommand(object):
             logger_str = log_str_factory(self.ran, call_args)
             self.log = Logger("command", logger_str)
 
-            self.log.info("starting process")
+            self.log.debug("starting process")
 
             if should_wait:
                 self._spawned_and_waited = True
@@ -774,19 +793,57 @@ class RunningCommand(object):
                 self.wait()
 
 
-    def wait(self):
+    def wait(self, timeout=None):
         """ waits for the running command to finish.  this is called on all
         running commands, eventually, except for ones that run in the background
+
+        if timeout is a number, it is the number of seconds to wait for the process to resolve. otherwise block on wait.
+
+        this function can raise a TimeoutException, either because of a `_timeout` on the command itself as it was
+        launched, or because of a timeout passed into this method.
         """
-        if not self._process_completed:
-            self._process_completed = True
+        if not self._waited_until_completion:
+
+            # if we've been given a timeout, we need to poll is_alive()
+            if timeout is not None:
+                waited_for = 0
+                sleep_amt = 0.1
+                if timeout < 0:
+                    raise RuntimeError("timeout cannot be negative")
+
+                # while we still have time to wait, run this loop
+                # notice that alive and exit_code are only defined in this loop, but the loop is also guaranteed to run,
+                # defining them, given the constraints that timeout is non-negative
+                while waited_for <= timeout:
+                    alive, exit_code = self.process.is_alive()
+
+                    # if we're alive, we need to wait some more, but let's sleep before we poll again
+                    if alive:
+                        time.sleep(sleep_amt)
+                        waited_for += sleep_amt
+
+                    # but if we're not alive, we're done waiting
+                    else:
+                        break
+
+                # if we've made it this far, and we're still alive, then it means we timed out waiting
+                if alive:
+                    raise TimeoutException(None, self.ran)
+
+                # if we didn't time out, we fall through and let the rest of the code handle exit_code.
+                # notice that we set _waited_until_completion here, only if we didn't time out. this allows us to
+                # re-wait again on timeout, if we catch the TimeoutException in the parent frame
+                self._waited_until_completion = True
+
+            else:
+                exit_code = self.process.wait()
+                self._waited_until_completion = True
 
-            exit_code = self.process.wait()
             if self.process.timed_out:
                 # if we timed out, our exit code represents a signal, which is
                 # negative, so let's make it positive to store in our
                 # TimeoutException
-                raise TimeoutException(-exit_code)
+                raise TimeoutException(-exit_code, self.ran)
 
             else:
                 self.handle_command_exit_code(exit_code)
@@ -798,10 +855,15 @@ class RunningCommand(object):
                 if self.process._stdin_process:
                     self.process._stdin_process.command.wait()
 
-        self.log.info("process completed")
+            self.log.debug("process completed")
         return self
 
 
+    def is_alive(self):
+         """ returns whether or not we're still alive. this call has side-effects on OProc """
+         return self.process.is_alive()[0]
+
+
     def handle_command_exit_code(self, code):
         """ here we determine if we had an exception, or an error code that we
         weren't expecting to see.  if we did, we create and raise an exception
@@ -854,7 +916,7 @@ class RunningCommand(object):
         # so the slight timeout allows for that.
         while True:
             try:
-                chunk = self.process._pipe_queue.get(True, 0.001)
+                chunk = self.process._pipe_queue.get(True, self.call_args["iter_poll_time"])
             except Empty:
                 if self.call_args["iter_noblock"]:
                     return errno.EWOULDBLOCK
@@ -955,15 +1017,15 @@ def get_prepend_stack():
     return tl._prepend_stack
 
 
-def special_kwarg_validator(kwargs, invalid_list):
-    s1 = set(kwargs.keys())
+def special_kwarg_validator(passed_kwargs, merged_kwargs, invalid_list):
+    s1 = set(passed_kwargs.keys())
     invalid_args = []
 
     for args in invalid_list:
 
         if callable(args):
             fn = args
-            ret = fn(kwargs)
+            ret = fn(passed_kwargs, merged_kwargs)
             invalid_args.extend(ret)
 
         else:
@@ -993,12 +1055,14 @@ def get_fileno(ob):
 
     return fileno
 
+def ob_is_fd_based(ob):
+    return get_fileno(ob) is not None
 
 def ob_is_tty(ob):
     """ checks if an object (like a file-like object) is a tty.  """
     fileno = get_fileno(ob)
     is_tty = False
-    if fileno:
+    if fileno is not None:
         is_tty = os.isatty(fileno)
     return is_tty
 
@@ -1011,19 +1075,41 @@ def ob_is_pipe(ob):
     return is_pipe
 
 
-def tty_in_validator(kwargs):
+def tty_in_validator(passed_kwargs, merged_kwargs):
+    # here we'll validate that people aren't randomly shotgun-debugging different tty options and hoping that they'll
+    # work, without understanding what they do
     pairs = (("tty_in", "in"), ("tty_out", "out"))
     invalid = []
     for tty, std in pairs:
-        if tty in kwargs and ob_is_tty(kwargs.get(std, None)):
+        if tty in passed_kwargs and ob_is_tty(passed_kwargs.get(std, None)):
             args = (tty, std)
-            error = "`_%s` is a TTY already, so so it doesn't make sense \
-to set up a TTY with `_%s`" % (std, tty)
+            error = "`_%s` is a TTY already, so so it doesn't make sense to set up a TTY with `_%s`" % (std, tty)
             invalid.append((args, error))
 
+    # if unify_ttys is set, then both tty_in and tty_out must both be True
+    if merged_kwargs["unify_ttys"] and not (merged_kwargs["tty_in"] and merged_kwargs["tty_out"]):
+        invalid.append((("unify_ttys", "tty_in", "tty_out"),
+            "`_tty_in` and `_tty_out` must both be True if `_unify_ttys` is True"))
+
+    return invalid
+
+def fg_validator(passed_kwargs, merged_kwargs):
+    """ fg is not valid with basically every other option """
+
+    invalid = []
+    msg = """\
+_fg is invalid with nearly every other option, see warning and workaround here:
+
+    https://amoffat.github.io/sh/sections/special_arguments.html#fg"""
+    whitelist = set(("env", "fg", "cwd"))
+    offending = set(passed_kwargs.keys()) - whitelist
+
+    if "fg" in passed_kwargs and offending:
+        invalid.append(("fg", msg))
     return invalid
 
-def bufsize_validator(kwargs):
+
+def bufsize_validator(passed_kwargs, merged_kwargs):
     """ a validator to prevent a user from saying that they want custom
     buffering when they're using an in/out object that will be os.dup'd to the
     process, and has its own buffering.  an example is a pipe or a tty.  it
@@ -1031,14 +1117,14 @@ def bufsize_validator(kwargs):
     controls this. """
     invalid = []
 
-    in_ob = kwargs.get("in", None)
-    out_ob = kwargs.get("out", None)
+    in_ob = passed_kwargs.get("in", None)
+    out_ob = passed_kwargs.get("out", None)
 
-    in_buf = kwargs.get("in_bufsize", None)
-    out_buf = kwargs.get("out_bufsize", None)
+    in_buf = passed_kwargs.get("in_bufsize", None)
+    out_buf = passed_kwargs.get("out_bufsize", None)
 
-    in_no_buf = ob_is_tty(in_ob) or ob_is_pipe(in_ob)
-    out_no_buf = ob_is_tty(out_ob) or ob_is_pipe(out_ob)
+    in_no_buf = ob_is_fd_based(in_ob)
+    out_no_buf = ob_is_fd_based(out_ob)
 
     err = "Can't specify an {target} bufsize if the {target} target is a pipe or TTY"
 
@@ -1051,6 +1137,28 @@ def bufsize_validator(kwargs):
     return invalid
 
 
+def env_validator(passed_kwargs, merged_kwargs):
+    """ a validator to check that env is a dictionary and that all environment variable 
+    keys and values are strings. Otherwise, we would exit with a confusing exit code 255. """
+    invalid = []
+
+    env = passed_kwargs.get("env", None)
+    if env is None:
+        return invalid
+
+    if not isinstance(env, dict):
+        invalid.append(("env", "env must be a dict. Got {!r}".format(env)))
+        return invalid
+
+    for k, v in passed_kwargs["env"].items():
+        if not isinstance(k, str):
+            invalid.append(("env", "env key {!r} must be a str".format(k)))
+        if not isinstance(v, str):
+            invalid.append(("env", "value {!r} of env key {!r} must be a str".format(v, k)))
+
+    return invalid
+
+
 class Command(object):
     """ represents an un-run system program, like "ls" or "cd".  because it
     represents the program itself (and not a running instance of it), it should
@@ -1101,6 +1209,8 @@ class Command(object):
         "piped": None,
         "iter": None,
         "iter_noblock": None,
+        # the amount of time to sleep between polling for the iter output queue
+        "iter_poll_time": 0.1,
         "ok_code": 0,
         "cwd": None,
 
@@ -1117,6 +1227,7 @@ class Command(object):
         # ssh is one of those programs
         "tty_in": False,
         "tty_out": True,
+        "unify_ttys": False,
 
         "encoding": DEFAULT_ENCODING,
         "decode_errors": "strict",
@@ -1167,21 +1278,27 @@ class Command(object):
         # a callable that produces a log message from an argument tuple of the
         # command and the args
         "log_msg": None,
+
+        # whether or not to close all inherited fds. typically, this should be True, as inheriting fds can be a security
+        # vulnerability
+        "close_fds": True,
+
+        # a whitelist of the integer fds to pass through to the child process. setting this forces close_fds to be True
+        "pass_fds": set(),
     }
 
     # this is a collection of validators to make sure the special kwargs make
     # sense
     _kwarg_validators = (
-        (("fg", "bg"), "Command can't be run in the foreground and background"),
-        (("fg", "err_to_out"), "Can't redirect STDERR in foreground mode"),
         (("err", "err_to_out"), "Stderr is already being redirected"),
         (("piped", "iter"), "You cannot iterate when this command is being piped"),
-        (("piped", "no_pipe"), "Using a pipe doesn't make sense if you've \
-disabled the pipe"),
-        (("no_out", "iter"), "You cannot iterate over output if there is no \
-output"),
+        (("piped", "no_pipe"), "Using a pipe doesn't make sense if you've disabled the pipe"),
+        (("no_out", "iter"), "You cannot iterate over output if there is no output"),
+        (("close_fds", "pass_fds"), "Passing `pass_fds` forces `close_fds` to be True"),
         tty_in_validator,
         bufsize_validator,
+        env_validator,
+        fg_validator,
     )
 
 
@@ -1248,8 +1365,9 @@ output"),
                 call_args[parg] = kwargs[key]
                 del kwargs[key]
 
-        invalid_kwargs = special_kwarg_validator(call_args,
-                Command._kwarg_validators)
+        merged_args = Command._call_args.copy()
+        merged_args.update(call_args)
+        invalid_kwargs = special_kwarg_validator(call_args, merged_args, Command._kwarg_validators)
 
         if invalid_kwargs:
             exc_msg = []
@@ -1396,12 +1514,16 @@ output"),
         # if we're running in foreground mode, we need to completely bypass
         # launching a RunningCommand and OProc and just do a spawn
         if call_args["fg"]:
+
             if call_args["env"] is None:
                 launch = lambda: os.spawnv(os.P_WAIT, cmd[0], cmd)
             else:
                 launch = lambda: os.spawnve(os.P_WAIT, cmd[0], cmd, call_args["env"])
 
-            exit_code = launch()
+            cwd = call_args["cwd"] or os.getcwd()
+            with pushd(cwd):
+                exit_code = launch()
+
             exc_class = get_exc_exit_code_would_raise(exit_code,
                     call_args["ok_code"], call_args["piped"])
             if exc_class:
@@ -1521,7 +1643,7 @@ def aggregate_keywords(keywords, sep, prefix, raw=False):
                 k = k.replace("_", "-")
 
             if v is True:
-                processed.append(encode("--" + k))
+                processed.append(encode(prefix + k))
             elif v is False:
                 pass
             elif sep is None or sep == " ":
@@ -1690,14 +1812,11 @@ class OProc(object):
     def __init__(self, command, parent_log, cmd, stdin, stdout, stderr,
             call_args, pipe, process_assign_lock):
         """
-            cmd is the full string that will be exec'd.  it includes the program
-            name and all its arguments
+            cmd is the full list of arguments that will be exec'd.  it includes the program name and all its arguments.
 
-            stdin, stdout, stderr are what the child will use for standard
-            input/output/err
+            stdin, stdout, stderr are what the child will use for standard input/output/err.
 
-            call_args is a mapping of all the special keyword arguments to apply
-            to the child process
+            call_args is a mapping of all the special keyword arguments to apply to the child process.
         """
         self.command = command
         self.call_args = call_args
@@ -1726,32 +1845,42 @@ class OProc(object):
         # file-like object that is a tty, for example `sys.stdin`, then, later
         # on in this constructor, we're going to skip out on setting up pipes
         # and pseudoterminals for those endpoints
-        stdin_is_tty_or_pipe = ob_is_tty(stdin) or ob_is_pipe(stdin)
-        stdout_is_tty_or_pipe = ob_is_tty(stdout) or ob_is_pipe(stdout)
-        stderr_is_tty_or_pipe = ob_is_tty(stderr) or ob_is_pipe(stderr)
+        stdin_is_fd_based = ob_is_fd_based(stdin)
+        stdout_is_fd_based = ob_is_fd_based(stdout)
+        stderr_is_fd_based = ob_is_fd_based(stderr)
 
         tee_out = ca["tee"] in (True, "out")
         tee_err = ca["tee"] == "err"
 
-        # if we're passing in a custom stdout/out/err value, we obviously have
-        # to force not using single_tty
-        custom_in_out_err = stdin or stdout or stderr
-
-        single_tty = (ca["tty_in"] and ca["tty_out"])\
-                and not custom_in_out_err
+        single_tty = ca["tty_in"] and ca["tty_out"] and ca["unify_ttys"]
 
         # this logic is a little convoluted, but basically this top-level
         # if/else is for consolidating input and output TTYs into a single
         # TTY.  this is the only way some secure programs like ssh will
         # output correctly (is if stdout and stdin are both the same TTY)
         if single_tty:
-            self._stdin_read_fd, self._stdin_write_fd = pty.openpty()
+            # master_fd, slave_fd = pty.openpty()
+            #
+            # Anything that is written on the master end is provided to the process on the slave end as though it was
+            # input typed on a terminal. -"man 7 pty"
+            #
+            # later, in the child process, we're going to do this, so keep it in mind:
+            #
+            #    os.dup2(self._stdin_child_fd, 0)
+            #    os.dup2(self._stdout_child_fd, 1)
+            #    os.dup2(self._stderr_child_fd, 2)
+            self._stdin_parent_fd, self._stdin_child_fd = pty.openpty()
 
-            self._stdout_read_fd = os.dup(self._stdin_read_fd)
-            self._stdout_write_fd = os.dup(self._stdin_write_fd)
+            # this makes our parent fds behave like a terminal. it says that the very same fd that we "type" to (for
+            # stdin) is the same one that we see output printed to (for stdout)
+            self._stdout_parent_fd = os.dup(self._stdin_parent_fd)
 
-            self._stderr_read_fd = os.dup(self._stdin_read_fd)
-            self._stderr_write_fd = os.dup(self._stdin_write_fd)
+            # this line is what makes stdout and stdin attached to the same pty. in other words the process will write
+            # to the same underlying fd as stdout as it uses to read from for stdin. this makes programs like ssh happy
+            self._stdout_child_fd = os.dup(self._stdin_child_fd)
+
+            self._stderr_parent_fd = os.dup(self._stdin_parent_fd)
+            self._stderr_child_fd = os.dup(self._stdin_child_fd)
 
         # do not consolidate stdin and stdout.  this is the most common use-
         # case
@@ -1759,32 +1888,31 @@ class OProc(object):
             # this check here is because we may be doing piping and so our stdin
             # might be an instance of OProc
             if isinstance(stdin, OProc) and stdin.call_args["piped"]:
-                self._stdin_write_fd = stdin._pipe_fd
-                self._stdin_read_fd = None
+                self._stdin_child_fd = stdin._pipe_fd
+                self._stdin_parent_fd = None
                 self._stdin_process = stdin
 
-            elif stdin_is_tty_or_pipe:
-                self._stdin_write_fd = os.dup(get_fileno(stdin))
-                self._stdin_read_fd = None
+            elif stdin_is_fd_based:
+                self._stdin_child_fd = os.dup(get_fileno(stdin))
+                self._stdin_parent_fd = None
 
             elif ca["tty_in"]:
-                self._stdin_read_fd, self._stdin_write_fd = pty.openpty()
+                self._stdin_parent_fd, self._stdin_child_fd = pty.openpty()
 
             # tty_in=False is the default
             else:
-                self._stdin_write_fd, self._stdin_read_fd = os.pipe()
-
+                self._stdin_child_fd, self._stdin_parent_fd = os.pipe()
 
-            if stdout_is_tty_or_pipe and not tee_out:
-                self._stdout_write_fd = os.dup(get_fileno(stdout))
-                self._stdout_read_fd = None
+            if stdout_is_fd_based and not tee_out:
+                self._stdout_child_fd = os.dup(get_fileno(stdout))
+                self._stdout_parent_fd = None
 
             # tty_out=True is the default
             elif ca["tty_out"]:
-                self._stdout_read_fd, self._stdout_write_fd = pty.openpty()
+                self._stdout_parent_fd, self._stdout_child_fd = pty.openpty()
 
             else:
-                self._stdout_read_fd, self._stdout_write_fd = os.pipe()
+                self._stdout_parent_fd, self._stdout_child_fd = os.pipe()
 
             # unless STDERR is going to STDOUT, it ALWAYS needs to be a pipe,
             # and never a PTY.  the reason for this is not totally clear to me,
@@ -1797,27 +1925,27 @@ class OProc(object):
                 # we should not specify a read_fd, because stdout is dup'd
                 # directly to the stdout fd (no pipe), and so stderr won't have
                 # a slave end of a pipe either to dup
-                if stdout_is_tty_or_pipe and not tee_out:
-                    self._stderr_read_fd = None
+                if stdout_is_fd_based and not tee_out:
+                    self._stderr_parent_fd = None
                 else:
-                    self._stderr_read_fd = os.dup(self._stdout_read_fd)
-                self._stderr_write_fd = os.dup(self._stdout_write_fd)
+                    self._stderr_parent_fd = os.dup(self._stdout_parent_fd)
+                self._stderr_child_fd = os.dup(self._stdout_child_fd)
 
 
-            elif stderr_is_tty_or_pipe and not tee_err:
-                self._stderr_write_fd = os.dup(get_fileno(stderr))
-                self._stderr_read_fd = None
+            elif stderr_is_fd_based and not tee_err:
+                self._stderr_child_fd = os.dup(get_fileno(stderr))
+                self._stderr_parent_fd = None
 
             else:
-                self._stderr_read_fd, self._stderr_write_fd = os.pipe()
+                self._stderr_parent_fd, self._stderr_child_fd = os.pipe()
 
 
         piped = ca["piped"]
         self._pipe_fd = None
         if piped:
-            fd_to_use = self._stdout_read_fd
+            fd_to_use = self._stdout_parent_fd
             if piped == "err":
-                fd_to_use = self._stderr_read_fd
+                fd_to_use = self._stderr_parent_fd
             self._pipe_fd = os.dup(fd_to_use)
 
 
@@ -1826,18 +1954,7 @@ class OProc(object):
 
         self.ctty = None
         if needs_ctty:
-            self.ctty = os.ttyname(self._stdin_write_fd)
-
-        # this is a hack, but what we're doing here is intentionally throwing an
-        # OSError exception if our child processes's directory doesn't exist,
-        # but we're doing it BEFORE we fork.  the reason for before the fork is
-        # error handling.  i'm currently too lazy to implement what
-        # subprocess.py did and set up a error pipe to handle exceptions that
-        # happen in the child between fork and exec.  it has only been seen in
-        # the wild for a missing cwd, so we'll handle it here.
-        cwd = ca["cwd"]
-        if cwd is not None and not os.path.exists(cwd):
-            os.chdir(cwd)
+            self.ctty = os.ttyname(self._stdin_child_fd)
 
         gc_enabled = gc.isenabled()
         if gc_enabled:
@@ -1847,12 +1964,12 @@ class OProc(object):
         session_pipe_read, session_pipe_write = os.pipe()
         exc_pipe_read, exc_pipe_write = os.pipe()
 
-        # this pipe is for synchronzing with the child that the parent has
+        # this pipe is for synchronizing with the child that the parent has
         # closed its in/out/err fds.  this is a bug on OSX (but not linux),
         # where we can lose output sometimes, due to a race, if we do
-        # os.close(self._stdout_write_fd) in the parent after the child starts
+        # os.close(self._stdout_child_fd) in the parent after the child starts
         # writing.
-        if IS_OSX:
+        if IS_MACOS:
             close_pipe_read, close_pipe_write = os.pipe()
 
         # session id, group id, process id
@@ -1862,11 +1979,24 @@ class OProc(object):
 
         # child
         if self.pid == 0: # pragma: no cover
-            if IS_OSX:
+            if IS_MACOS:
                 os.read(close_pipe_read, 1)
                 os.close(close_pipe_read)
                 os.close(close_pipe_write)
 
+            # this is critical
+            # our exc_pipe_write must have CLOEXEC enabled. the reason for this is tricky:
+            # if our child (the block we're in now), has an exception, we need to be able to write to exc_pipe_write, so
+            # that when the parent does os.read(exc_pipe_read), it gets our traceback.  however, os.read(exc_pipe_read)
+            # in the parent blocks, so if our child *doesn't* have an exception, and doesn't close the writing end, it
+            # hangs forever.  not good!  but obviously the child can't close the writing end until it knows it's not
+            # going to have an exception, which is impossible to know because but what if os.execv has an exception?  so
+            # the answer is CLOEXEC, so that the writing end of the pipe gets closed upon successful exec, and the
+            # parent reading the read end won't block (close breaks the block).
+            flags = fcntl.fcntl(exc_pipe_write, fcntl.F_GETFD)
+            flags |= fcntl.FD_CLOEXEC
+            fcntl.fcntl(exc_pipe_write, fcntl.F_SETFD, flags) 
+
             try:
                 # ignoring SIGHUP lets us persist even after the parent process
                 # exits.  only ignore if we're backgrounded
@@ -1900,7 +2030,7 @@ class OProc(object):
                 payload = ("%d,%d" % (sid, pgid)).encode(DEFAULT_ENCODING)
                 os.write(session_pipe_write, payload)
 
-                if ca["tty_out"] and not stdout_is_tty_or_pipe and not single_tty:
+                if ca["tty_out"] and not stdout_is_fd_based and not single_tty:
                     # set raw mode, so there isn't any weird translation of
                     # newlines to \r\n and other oddities.  we're not outputting
                     # to a terminal anyways
@@ -1908,29 +2038,30 @@ class OProc(object):
                     # we HAVE to do this here, and not in the parent process,
                     # because we have to guarantee that this is set before the
                     # child process is run, and we can't do it twice.
-                    tty.setraw(self._stdout_write_fd)
+                    tty.setraw(self._stdout_child_fd)
 
 
                 # if the parent-side fd for stdin exists, close it.  the case
                 # where it may not exist is if we're using piping
-                if self._stdin_read_fd:
-                    os.close(self._stdin_read_fd)
+                if self._stdin_parent_fd:
+                    os.close(self._stdin_parent_fd)
 
-                if self._stdout_read_fd:
-                    os.close(self._stdout_read_fd)
+                if self._stdout_parent_fd:
+                    os.close(self._stdout_parent_fd)
 
-                if self._stderr_read_fd:
-                    os.close(self._stderr_read_fd)
+                if self._stderr_parent_fd:
+                    os.close(self._stderr_parent_fd)
 
                 os.close(session_pipe_read)
                 os.close(exc_pipe_read)
 
+                cwd = ca["cwd"]
                 if cwd:
                     os.chdir(cwd)
 
-                os.dup2(self._stdin_write_fd, 0)
-                os.dup2(self._stdout_write_fd, 1)
-                os.dup2(self._stderr_write_fd, 2)
+                os.dup2(self._stdin_child_fd, 0)
+                os.dup2(self._stdout_child_fd, 1)
+                os.dup2(self._stderr_child_fd, 2)
 
 
                 # set our controlling terminal, but only if we're using a tty
@@ -1939,7 +2070,7 @@ class OProc(object):
                     tmp_fd = os.open(os.ttyname(0), os.O_RDWR)
                     os.close(tmp_fd)
 
-                if ca["tty_out"] and not stdout_is_tty_or_pipe:
+                if ca["tty_out"] and not stdout_is_fd_based:
                     setwinsize(1, ca["tty_size"])
 
                 if ca["uid"] is not None:
@@ -1950,10 +2081,22 @@ class OProc(object):
                 if callable(preexec_fn):
                     preexec_fn()
 
+                close_fds = ca["close_fds"]
+                if ca["pass_fds"]:
+                    close_fds = True
 
-                # don't inherit file descriptors
-                max_fd = resource.getrlimit(resource.RLIMIT_NOFILE)[0]
-                os.closerange(3, max_fd)
+                if close_fds:
+                    pass_fds = set((0, 1, 2, exc_pipe_write))
+                    pass_fds.update(ca["pass_fds"])
+
+                    # don't inherit file descriptors
+                    inherited_fds = os.listdir("/dev/fd")
+                    inherited_fds = set(int(fd) for fd in inherited_fds) - pass_fds
+                    for fd in inherited_fds:
+                        try:
+                            os.close(fd)
+                        except OSError:
+                            pass
 
                 # actually execute the process
                 if ca["env"] is None:
@@ -1970,10 +2113,15 @@ class OProc(object):
             # and the exec.  this should be reported.
             except:
                 # some helpful debugging
+                tb = traceback.format_exc().encode("utf8", "ignore")
+
                 try:
-                    tb = traceback.format_exc().encode("utf8", "ignore")
                     os.write(exc_pipe_write, tb)
 
+                except Exception as e:
+                    # dump to stderr if we cannot save it to exc_pipe_write
+                    sys.stderr.write("\nFATAL SH ERROR: %s\n" % e)
+
                 finally:
                     os._exit(255)
 
@@ -1982,14 +2130,14 @@ class OProc(object):
             if gc_enabled:
                 gc.enable()
 
-            os.close(self._stdin_write_fd)
-            os.close(self._stdout_write_fd)
-            os.close(self._stderr_write_fd)
+            os.close(self._stdin_child_fd)
+            os.close(self._stdout_child_fd)
+            os.close(self._stderr_child_fd)
 
             # tell our child process that we've closed our write_fds, so it is
             # ok to proceed towards exec.  see the comment where this pipe is
             # opened, for why this is necessary
-            if IS_OSX:
+            if IS_MACOS:
                 os.close(close_pipe_read)
                 os.write(close_pipe_write, str(1).encode(DEFAULT_ENCODING))
                 os.close(close_pipe_write)
@@ -2020,7 +2168,12 @@ class OProc(object):
             # to prevent race conditions
             self.exit_code = None
 
-            self.stdin = stdin or Queue()
+            self.stdin = stdin
+
+            # this accounts for when _out is a callable that is passed stdin.  in that case, if stdin is unspecified, we
+            # must set it to a queue, so callbacks can put things on it
+            if callable(ca["out"]) and self.stdin is None:
+                self.stdin = Queue()
 
             # _pipe_queue is used internally to hand off stdout from one process
             # to another.  by default, all stdout from a process gets dumped
@@ -2038,8 +2191,8 @@ class OProc(object):
             self._stdout = deque(maxlen=ca["internal_bufsize"])
             self._stderr = deque(maxlen=ca["internal_bufsize"])
 
-            if ca["tty_in"] and not stdin_is_tty_or_pipe:
-                setwinsize(self._stdin_read_fd, ca["tty_size"])
+            if ca["tty_in"] and not stdin_is_fd_based:
+                setwinsize(self._stdin_parent_fd, ca["tty_size"])
 
 
             self.log = parent_log.get_child("process", repr(self))
@@ -2048,22 +2201,17 @@ class OProc(object):
             self.log.debug("started process")
 
             # disable echoing, but only if it's a tty that we created ourselves
-            if ca["tty_in"] and not stdin_is_tty_or_pipe:
-                attr = termios.tcgetattr(self._stdin_read_fd)
+            if ca["tty_in"] and not stdin_is_fd_based:
+                attr = termios.tcgetattr(self._stdin_parent_fd)
                 attr[3] &= ~termios.ECHO
-                termios.tcsetattr(self._stdin_read_fd, termios.TCSANOW, attr)
-
-            # we're only going to create a stdin thread iff we have potential
-            # for stdin to come in.  this would be through a stdout callback or
-            # through an object we've passed in for stdin
-            potentially_has_input = callable(stdout) or stdin
+                termios.tcsetattr(self._stdin_parent_fd, termios.TCSANOW, attr)
 
             # this represents the connection from a Queue object (or whatever
             # we're using to feed STDIN) to the process's STDIN fd
             self._stdin_stream = None
-            if self._stdin_read_fd and potentially_has_input:
+            if self._stdin_parent_fd:
                 log = self.log.get_child("streamwriter", "stdin")
-                self._stdin_stream =  StreamWriter(log, self._stdin_read_fd,
+                self._stdin_stream =  StreamWriter(log, self._stdin_parent_fd,
                         self.stdin, ca["in_bufsize"], ca["encoding"],
                         ca["tty_in"])
 
@@ -2088,20 +2236,19 @@ class OProc(object):
             # already hooked up this processes's stdout fd to the other
             # processes's stdin fd
             self._stdout_stream = None
-            if not pipe_out and self._stdout_read_fd:
+            if not pipe_out and self._stdout_parent_fd:
                 if callable(stdout):
                     stdout = construct_streamreader_callback(self, stdout)
                 self._stdout_stream = \
                         StreamReader(
                                 self.log.get_child("streamreader", "stdout"),
-                                self._stdout_read_fd, stdout, self._stdout,
+                                self._stdout_parent_fd, stdout, self._stdout,
                                 ca["out_bufsize"], ca["encoding"],
                                 ca["decode_errors"], stdout_pipe,
                                 save_data=save_stdout)
 
-            elif self._stdout_read_fd:
-                os.close(self._stdout_read_fd)
-
+            elif self._stdout_parent_fd:
+                os.close(self._stdout_parent_fd)
 
             # if stderr is going to one place (because it's grouped with stdout,
             # or we're dealing with a single tty), then we don't actually need a
@@ -2109,7 +2256,7 @@ class OProc(object):
             # stdout above
             self._stderr_stream = None
             if stderr is not OProc.STDOUT and not single_tty and not pipe_err \
-                    and self._stderr_read_fd:
+                    and self._stderr_parent_fd:
 
                 stderr_pipe = None
                 if pipe is OProc.STDERR and not ca["no_pipe"]:
@@ -2122,12 +2269,12 @@ class OProc(object):
                     stderr = construct_streamreader_callback(self, stderr)
 
                 self._stderr_stream = StreamReader(Logger("streamreader"),
-                        self._stderr_read_fd, stderr, self._stderr,
+                        self._stderr_parent_fd, stderr, self._stderr,
                         ca["err_bufsize"], ca["encoding"], ca["decode_errors"],
                         stderr_pipe, save_data=save_stderr)
 
-            elif self._stderr_read_fd:
-                os.close(self._stderr_read_fd)
+            elif self._stderr_parent_fd:
+                os.close(self._stderr_parent_fd)
 
 
             def timeout_fn():
@@ -2250,13 +2397,13 @@ class OProc(object):
 
     def get_pgid(self):
         """ return the CURRENT group id of the process. this differs from
-        self.pgid in that this refects the current state of the process, where
+        self.pgid in that this reflects the current state of the process, where
         self.pgid is the group id at launch """
         return os.getpgid(self.pid)
 
     def get_sid(self):
         """ return the CURRENT session id of the process. this differs from
-        self.sid in that this refects the current state of the process, where
+        self.sid in that this reflects the current state of the process, where
         self.sid is the session id at launch """
         return os.getsid(self.pid)
 
@@ -2339,8 +2486,8 @@ class OProc(object):
         # the CTTY, and closing it prematurely will send a SIGHUP.  we also
         # don't want to close it if there's a self._stdin_stream, because that
         # is in charge of closing it also
-        if self._stdin_read_fd and not self._stdin_stream:
-            os.close(self._stdin_read_fd)
+        if self._stdin_parent_fd and not self._stdin_stream:
+            os.close(self._stdin_parent_fd)
 
 
     def wait(self):
@@ -2553,6 +2700,12 @@ def determine_how_to_read_input(input_obj):
         log_msg = "generator"
         get_chunk = get_iter_chunk_reader(iter(input_obj))
 
+    elif input_obj is None:
+        log_msg = "None"
+        def raise_():
+            raise DoneReadingForever
+        get_chunk = raise_
+
     else:
         try:
             it = iter(input_obj)
@@ -3183,9 +3336,8 @@ class Environment(dict):
 
         # somebody tried to be funny and do "from sh import *"
         if k == "__all__":
-            raise RuntimeError("Cannot import * from sh. \
-Please import sh or import programs individually.")
-
+            warnings.warn("Cannot import * from sh. Please import sh or import programs individually.")
+            return []
 
         # check if we're naming a dynamically generated ReturnCode exception
         exc = get_exc_from_name(k)
@@ -3273,6 +3425,7 @@ def git(orig): # pragma: no cover
     cmd = orig.bake(_tty_out=False)
     return cmd
 
+
 @contrib("sudo")
 def sudo(orig): # pragma: no cover
     """ a nicer version of sudo that uses getpass to ask for a password, or
@@ -3300,6 +3453,85 @@ def sudo(orig): # pragma: no cover
     return cmd
 
 
+@contrib("ssh")
+def ssh(orig): # pragma: no cover
+    """ An ssh command for automatic password login """
+
+    class SessionContent(object):
+        def __init__(self):
+            self.chars = deque(maxlen=50000)
+            self.lines = deque(maxlen=5000)
+            self.line_chars = []
+            self.last_line = ""
+            self.cur_char = ""
+
+        def append_char(self, char):
+            if char == "\n":
+                line = self.cur_line
+                self.last_line = line
+                self.lines.append(line)
+                self.line_chars = []
+            else:
+                self.line_chars.append(char)
+
+            self.chars.append(char)
+            self.cur_char = char
+
+        @property
+        def cur_line(self):
+            line = "".join(self.line_chars)
+            return line
+
+    class SSHInteract(object):
+        def __init__(self, prompt_match, pass_getter, out_handler, login_success):
+            self.prompt_match = prompt_match
+            self.pass_getter = pass_getter
+            self.out_handler = out_handler
+            self.login_success = login_success
+            self.content = SessionContent()
+
+            # some basic state
+            self.pw_entered = False
+            self.success = False
+
+        def __call__(self, char, stdin):
+            self.content.append_char(char)
+
+            if self.pw_entered and not self.success:
+                self.success = self.login_success(self.content)
+
+            if self.success:
+                return self.out_handler(self.content, stdin)
+
+            if self.prompt_match(self.content):
+                password = self.pass_getter()
+                stdin.put(password + "\n")
+                self.pw_entered = True
+
+
+    def process(args, kwargs):
+        real_out_handler = kwargs.pop("interact")
+        password = kwargs.pop("password", None)
+        login_success = kwargs.pop("login_success", None)
+        prompt_match = kwargs.pop("prompt", None)
+        prompt = "Please enter SSH password: "
+
+        if prompt_match is None:
+            prompt_match = lambda content: content.cur_line.endswith("password: ")
+
+        if password is None:
+            pass_getter = lambda: getpass.getpass(prompt=prompt)
+        else:
+            pass_getter = lambda: password.rstrip("\n")
+
+        if login_success is None:
+            login_success = lambda content: True
+
+        kwargs["_out"] = SSHInteract(prompt_match, pass_getter, real_out_handler, login_success)
+        return args, kwargs
+
+    cmd = orig.bake(_out_bufsize=0, _tty_in=True, _unify_ttys=True, _arg_preprocess=process)
+    return cmd
 
 
 def run_repl(env): # pragma: no cover
@@ -3364,12 +3596,32 @@ class SelfWrapper(ModuleType):
         # cached module from sys.modules.  if we don't, it gets re-used, and any
         # old baked params get used, which is not what we want
         parent = inspect.stack()[1]
-        code = parent[4][0].strip()
-        parsed = ast.parse(code)
-        module_name = parsed.body[0].targets[0].id
+        try:
+            code = parent[4][0].strip()
+        except TypeError:
+            # On the REPL or from the commandline, we don't get the source code in the
+            # top stack frame
+            # Older versions of pypy don't set parent[1] the same way as CPython or newer versions
+            # of Pypy so we have to special case that too.
+            if parent[1] in ('<stdin>', '<string>') or (
+                    parent[1] == '<module>' and platform.python_implementation().lower() == 'pypy'):
+                # This depends on things like Python's calling convention and the layout of stack
+                # frames but it's a fix for a bug in a very cornery cornercase so....
+                module_name = parent[0].f_code.co_names[-1]
+            else:
+                raise
+        else:
+            parsed = ast.parse(code)
+            try:
+                module_name = parsed.body[0].targets[0].id
+            except Exception:
+                # Diagnose what went wrong
+                if not isinstance(parsed.body[0], ast.Assign):
+                    raise RuntimeError("A new execution context must be assigned to a variable")
+                raise
 
         if module_name == __name__:
-            raise RuntimeError("Cannot use the name 'sh' as an execution context")
+            raise RuntimeError("Cannot use the name '%s' as an execution context" % __name__)
 
         sys.modules.pop(module_name, None)
 
@@ -3391,12 +3643,17 @@ def register_importer():
     """
 
     def test(importer):
-        return importer.__class__.__name__ == ModuleImporterFromVariables.__name__
+        try:
+            return importer.__class__.__name__ == ModuleImporterFromVariables.__name__
+        except AttributeError:
+            # ran into importer which is not a class instance
+            return False
+
     already_registered = any([True for i in sys.meta_path if test(i)])
 
     if not already_registered:
         importer = ModuleImporterFromVariables(
-            restrict_to=["SelfWrapper"],
+            restrict_to=[SelfWrapper.__name__],
         )
         sys.meta_path.insert(0, importer)
 
@@ -3426,13 +3683,21 @@ class ModuleImporterFromVariables(object):
             derp = sh()
             from derp import ls
 
-        here, mod_fullname will be "derp".  keep that in mind as we go throug
+        here, mod_fullname will be "derp".  keep that in mind as we go through
         the rest of this function """
 
         parent_frame = inspect.currentframe().f_back
-        while in_importlib(parent_frame):
+
+        while parent_frame and in_importlib(parent_frame):
             parent_frame = parent_frame.f_back
 
+        # Calling PyImport_ImportModule("some_module"); via the C API may not
+        # have a parent frame. Early-out to avoid in_importlib() trying to
+        # get f_code from None when looking for 'some_module'.
+        # This also happens when using gevent apparently.
+        if not parent_frame:
+            return None
+
         # this line is saying "hey, does mod_fullname exist as a name we've
         # defind previously?"  the purpose of this is to ensure that
         # mod_fullname is really a thing we've defined.  if we haven't defined
@@ -3457,7 +3722,7 @@ class ModuleImporterFromVariables(object):
         module = fetch_module_from_frame(mod_fullname, parent_frame)
 
         # we HAVE to include the module in sys.modules, per the import PEP.
-        # older verions of python were more lenient about this being set, but
+        # older versions of python were more lenient about this being set, but
         # not in >= python3.3, unfortunately.  this requirement necessitates the
         # ugly code in SelfWrapper.__call__
         sys.modules[mod_fullname] = module
@@ -3488,6 +3753,7 @@ def run_tests(env, locale, args, version, force_select, **extra_env): # pragma:
             env[k] = str(v)
 
         cmd = [py_bin, "-W", "ignore", os.path.join(THIS_DIR, "test.py")] + args[1:]
+        print("Running %r" % cmd)
         launch = lambda: os.spawnve(os.P_WAIT, cmd[0], cmd, env)
         return_code = launch()
 
@@ -3517,7 +3783,7 @@ if __name__ == "__main__": # pragma: no cover
     if args:
         action = args[0]
 
-    if action in ("test", "travis"):
+    if action in ("test", "travis", "tox"):
         import test
         coverage = None
         if test.HAS_UNICODE_LITERAL:
@@ -3530,12 +3796,11 @@ if __name__ == "__main__": # pragma: no cover
 
         # if we're testing locally, run all versions of python on the system
         if action == "test":
-            all_versions = ("2.6", "2.7", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6")
+            all_versions = ("2.6", "2.7", "3.1", "3.2", "3.3", "3.4", "3.5", "3.6", "3.7", "3.8")
 
-        # if we're testing on travis, just use the system's default python,
-        # since travis will spawn a vm per python version in our .travis.yml
-        # file
-        elif action == "travis":
+        # if we're testing on travis or tox, just use the system's default python, since travis will spawn a vm per
+        # python version in our .travis.yml file, and tox will run its matrix via tox.ini
+        elif action in ("travis", "tox"):
             v = sys.version_info
             sys_ver = "%d.%d" % (v[0], v[1])
             all_versions = (sys_ver,)
@@ -3546,17 +3811,21 @@ if __name__ == "__main__": # pragma: no cover
 
         all_locales = ("en_US.UTF-8", "C")
         i = 0
+        ran_versions = set()
         for locale in all_locales:
+            # make sure this locale is allowed
             if constrain_locales and locale not in constrain_locales:
                 continue
 
             for version in all_versions:
+                # make sure this version is allowed
                 if constrain_versions and version not in constrain_versions:
                     continue
 
                 for force_select in all_force_select:
                     env_copy = env.copy()
 
+                    ran_versions.add(version)
                     exit_code = run_tests(env_copy, locale, args, version,
                             force_select, SH_TEST_RUN_IDX=i)
 
@@ -3569,8 +3838,7 @@ if __name__ == "__main__": # pragma: no cover
 
                     i += 1
 
-        ran_versions = ",".join(all_versions)
-        print("Tested Python versions: %s" % ran_versions)
+        print("Tested Python versions: %s" % ",".join(sorted(list(ran_versions))))
 
     else:
         env = Environment(globals())
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 0000000..1f986c7
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,13 @@
+[tox]
+# virtualenv for py26 is broken, so don't put it here
+envlist = py{27,31,32,33,34,35,36,37,38},docs
+
+[testenv]
+deps = -r requirements-dev.txt
+commands =
+    python sh.py tox
+
+[testenv:docs]
+basepython = python3
+commands =
+    python setup.py check --restructuredtext --metadata --strict


Test patch:
diff --git a/docker_test_suite/Dockerfile b/docker_test_suite/Dockerfile
new file mode 100644
index 0000000..d169ecf
--- /dev/null
+++ b/docker_test_suite/Dockerfile
@@ -0,0 +1,47 @@
+FROM ubuntu:bionic
+
+ARG cache_bust
+RUN apt-get update
+RUN apt-get -y install locales
+
+RUN locale-gen en_US.UTF-8  
+ENV LANG en_US.UTF-8  
+ENV LANGUAGE en_US:en  
+ENV LC_ALL en_US.UTF-8 
+
+RUN apt-get -y install\
+    software-properties-common\
+    curl\
+    sudo\
+    python\
+    lsof
+
+RUN add-apt-repository ppa:deadsnakes/ppa
+RUN apt-get update
+RUN apt-get -y install\
+    python2.6\
+    python2.7\
+    python3.1\
+    python3.2\
+    python3.3\
+    python3.4\
+    python3.5\
+    python3.6\
+    python3.7\
+    python3.8
+
+RUN apt-get -y install python3-distutils\
+    && curl https://bootstrap.pypa.io/get-pip.py | python -
+
+ARG uid=1000
+RUN groupadd -g $uid shtest\
+    && useradd -m -u $uid -g $uid shtest\
+    && gpasswd -a shtest sudo\
+    && echo "shtest:shtest" | chpasswd
+
+COPY requirements-dev.txt /tmp/
+RUN pip install -r /tmp/requirements-dev.txt
+
+USER shtest
+WORKDIR /home/shtest/sh
+ENTRYPOINT ["python", "sh.py", "test"]
diff --git a/docker_test_suite/build.sh b/docker_test_suite/build.sh
new file mode 100755
index 0000000..6ed7ec4
--- /dev/null
+++ b/docker_test_suite/build.sh
@@ -0,0 +1,4 @@
+#!/bin/bash
+set -ex
+cp ../requirements-dev.txt .
+docker build -t amoffat/shtest $@ .
diff --git a/docker_test_suite/run.sh b/docker_test_suite/run.sh
new file mode 100755
index 0000000..bd92b85
--- /dev/null
+++ b/docker_test_suite/run.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+set -ex
+docker run -it --rm -v $(pwd)/../:/home/shtest/sh amoffat/shtest $@
diff --git a/docker_test_suite/shell.sh b/docker_test_suite/shell.sh
new file mode 100755
index 0000000..234e8e0
--- /dev/null
+++ b/docker_test_suite/shell.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+set -ex
+docker run -it --rm -v $(pwd)/../:/home/shtest/sh --entrypoint=/bin/bash amoffat/shtest $@
diff --git a/test.py b/test.py
index 68ef40c..c790409 100644
--- a/test.py
+++ b/test.py
@@ -15,20 +15,23 @@ if HAS_UNICODE_LITERAL:
     run_idx = int(os.environ.pop("SH_TEST_RUN_IDX", "0"))
     first_run = run_idx == 0
 
-    import coverage
-
-    # for some reason, we can't run auto_data on the first run, or the coverage
-    # numbers get really screwed up
-    auto_data = True
-    if first_run:
-        auto_data = False
+    try:
+        import coverage
+    except ImportError:
+        pass
+    else:
+        # for some reason, we can't run auto_data on the first run, or the coverage
+        # numbers get really screwed up
+        auto_data = True
+        if first_run:
+            auto_data = False
 
-    cov = coverage.Coverage(auto_data=auto_data)
+        cov = coverage.Coverage(auto_data=auto_data)
 
-    if first_run:
-        cov.erase()
+        if first_run:
+            cov.erase()
 
-    cov.start()
+        cov.start()
 
 
 from os.path import exists, join, realpath, dirname, split
@@ -50,6 +53,7 @@ import sh
 import signal
 import errno
 import stat
+import fcntl
 import platform
 from functools import wraps
 import time
@@ -58,7 +62,7 @@ import inspect
 # we have to use the real path because on osx, /tmp is a symlink to
 # /private/tmp, and so assertions that gettempdir() == sh.pwd() will fail
 tempdir = realpath(tempfile.gettempdir())
-IS_OSX = platform.system() == "Darwin"
+IS_MACOS = platform.system() in ("AIX", "Darwin")
 
 
 # these 3 functions are helpers for modifying PYTHONPATH with a module's main
@@ -93,22 +97,21 @@ if IS_PY3:
     ioStringIO = StringIO
     from io import BytesIO as cStringIO
     iocStringIO = cStringIO
-    python = sh.Command(sh.which("python%d.%d" % sys.version_info[:2]))
 else:
     from StringIO import StringIO
     from cStringIO import StringIO as cStringIO
     from io import StringIO as ioStringIO
     from io import BytesIO as iocStringIO
-    python = sh.python
 
 THIS_DIR = dirname(os.path.abspath(__file__))
 
+system_python = sh.Command(sys.executable)
 
 # this is to ensure that our `python` helper here is able to import our local sh
 # module, and not the system one
 baked_env = os.environ.copy()
 append_module_path(baked_env, sh)
-python = python.bake(_env=baked_env)
+python = system_python.bake(_env=baked_env)
 
 
 if hasattr(logging, 'NullHandler'):
@@ -154,7 +157,7 @@ def requires_progs(*progs):
 
 requires_posix = skipUnless(os.name == "posix", "Requires POSIX")
 requires_utf8 = skipUnless(sh.DEFAULT_ENCODING == "UTF-8", "System encoding must be UTF-8")
-not_osx = skipUnless(not IS_OSX, "Doesn't work on OSX")
+not_macos = skipUnless(not IS_MACOS, "Doesn't work on MacOS")
 requires_py3 = skipUnless(IS_PY3, "Test only works on Python 3")
 requires_py35 = skipUnless(IS_PY3 and MINOR_VER >= 5, "Test only works on Python 3.5 or higher")
 
@@ -209,8 +212,8 @@ class BaseTests(unittest.TestCase):
         with warnings.catch_warnings(record=True) as w:
             fn(*args, **kwargs)
 
-            assert len(w) == 1
-            assert issubclass(w[-1].category, DeprecationWarning)
+            self.assertEqual(len(w), 1)
+            self.assertTrue(issubclass(w[-1].category, DeprecationWarning))
 
 
 @requires_posix
@@ -245,16 +248,15 @@ class FunctionalTests(BaseTests):
         py = create_tmp_test("exit(1)")
 
         arg = "漢字"
+        native_arg = arg
         if not IS_PY3:
             arg = arg.decode("utf8")
 
+
         try:
             python(py.name, arg, _encoding="utf8")
         except ErrorReturnCode as e:
-            if IS_PY3:
-                self.assertTrue(arg in str(e))
-            else:
-                self.assertTrue(arg in unicode(e))
+            self.assertTrue(native_arg in str(e))
         else:
             self.fail("exception wasn't raised")
 
@@ -285,6 +287,17 @@ print(args[0])
         out = python(py.name, 3).strip()
         self.assertEqual(out, "3")
 
+    def test_empty_stdin_no_hang(self):
+        py = create_tmp_test("""
+import sys
+data = sys.stdin.read()
+sys.stdout.write("no hang")
+""")
+        out = python(py.name, _in="", _timeout=2)
+        self.assertEqual(out, "no hang")
+
+        out = python(py.name, _in=None, _timeout=2)
+        self.assertEqual(out, "no hang")
 
     def test_exit_code(self):
         from sh import ErrorReturnCode
@@ -295,7 +308,7 @@ exit(3)
 
     def test_patched_glob(self):
         from glob import glob
-        
+
         py = create_tmp_test("""
 import sys
 print(sys.argv[1:])
@@ -357,7 +370,7 @@ exit(3)
 
         exc_to_test = ErrorReturnCode_2
         code_to_pass = 2
-        if IS_OSX:
+        if IS_MACOS:
             exc_to_test = ErrorReturnCode_1
             code_to_pass = 1
         self.assertRaises(exc_to_test, ls, "/aofwje/garogjao4a/eoan3on")
@@ -396,7 +409,6 @@ print(args)
         self.assertEqual(out, "[\"one two's three\"]")
 
     def test_multiple_pipes(self):
-        from sh import tr, python
         import time
 
         py = create_tmp_test("""
@@ -506,41 +518,30 @@ while True:
         # this is the environment we'll pass into our commands
         env = {"HERP": "DERP"}
 
-        # python on osx will bizarrely add some extra environment variables that
-        # i didn't ask for.  for this test, we prune those out if they exist
-        osx_cruft = [
-            "__CF_USER_TEXT_ENCODING",
-            "__PYVENV_LAUNCHER__",
-            "VERSIONER_PYTHON_PREFER_32_BIT",
-            "VERSIONER_PYTHON_VERSION",
-        ]
-
         # first we test that the environment exists in our child process as
         # we've set it
         py = create_tmp_test("""
 import os
 
-osx_cruft = %s
-for key in osx_cruft:
-    try: del os.environ[key]
-    except: pass
-print(os.environ["HERP"] + " " + str(len(os.environ)))
-""" % osx_cruft)
+for key in list(os.environ.keys()):
+    if key != "HERP":
+        del os.environ[key]
+print(dict(os.environ))
+""")
         out = python(py.name, _env=env).strip()
-        self.assertEqual(out, "DERP 1")
+        self.assertEqual(out, "{'HERP': 'DERP'}")
 
         py = create_tmp_test("""
 import os, sys
 sys.path.insert(0, os.getcwd())
 import sh
-osx_cruft = %s
-for key in osx_cruft:
-    try: del os.environ[key]
-    except: pass
-print(sh.HERP + " " + str(len(os.environ)))
-""" % osx_cruft)
+for key in list(os.environ.keys()):
+    if key != "HERP":
+        del os.environ[key]
+print(dict(HERP=sh.HERP))
+""")
         out = python(py.name, _env=env, _cwd=THIS_DIR).strip()
-        self.assertEqual(out, "DERP 1")
+        self.assertEqual(out, "{'HERP': 'DERP'}")
 
 
     def test_which(self):
@@ -563,6 +564,78 @@ print("hi")
         found_path = which(test_name, [test_path])
         self.assertEqual(found_path, py.name)
 
+    def test_no_close_fds(self):
+        # guarantee some extra fds in our parent process that don't close on exec.  we have to explicitly do this
+        # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent
+        # security vulnerabilities.  python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s
+        #
+        # https://www.python.org/dev/peps/pep-0446/
+        tmp = [tempfile.TemporaryFile() for i in range(10)]
+        for t in tmp:
+            flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD)
+            flags &= ~fcntl.FD_CLOEXEC
+            fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) 
+        first_fd = tmp[0].fileno()
+
+        py = create_tmp_test("""
+import os
+print(len(os.listdir("/dev/fd")))
+""")
+        out = python(py.name, _close_fds=False).strip()
+        # pick some number greater than 4, since it's hard to know exactly how many fds will be open/inherted in the
+        # child
+        self.assertTrue(int(out) > 7)
+
+        for t in tmp:
+            t.close()
+
+    def test_close_fds(self):
+        # guarantee some extra fds in our parent process that don't close on exec.  we have to explicitly do this
+        # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent
+        # security vulnerabilities.  python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s
+        #
+        # https://www.python.org/dev/peps/pep-0446/
+        tmp = [tempfile.TemporaryFile() for i in range(10)]
+        for t in tmp:
+            flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD)
+            flags &= ~fcntl.FD_CLOEXEC
+            fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) 
+
+        py = create_tmp_test("""
+import os
+print(os.listdir("/dev/fd"))
+""")
+        out = python(py.name).strip()
+        self.assertEqual(out, "['0', '1', '2', '3']")
+
+        for t in tmp:
+            t.close()
+
+
+    def test_pass_fds(self):
+        # guarantee some extra fds in our parent process that don't close on exec.  we have to explicitly do this
+        # because at some point (I believe python 3.4), python started being more stringent with closing fds to prevent
+        # security vulnerabilities.  python 2.7, for example, doesn't set CLOEXEC on tempfile.TemporaryFile()s
+        #
+        # https://www.python.org/dev/peps/pep-0446/
+        tmp = [tempfile.TemporaryFile() for i in range(10)]
+        for t in tmp:
+            flags = fcntl.fcntl(t.fileno(), fcntl.F_GETFD)
+            flags &= ~fcntl.FD_CLOEXEC
+            fcntl.fcntl(t.fileno(), fcntl.F_SETFD, flags) 
+        last_fd = tmp[-1].fileno()
+
+        py = create_tmp_test("""
+import os
+print(os.listdir("/dev/fd"))
+""")
+        out = python(py.name, _pass_fds=[last_fd]).strip()
+        inherited = [0, 1, 2, 3, last_fd]
+        inherited_str = [str(i) for i in inherited]
+        self.assertEqual(out, str(inherited_str))
+
+        for t in tmp:
+            t.close()
 
     def test_no_arg(self):
         import pwd
@@ -576,6 +649,17 @@ print("hi")
         self.assertRaises(TypeError, ls, _iter=True, _piped=True)
 
 
+    def test_invalid_env(self):
+        from sh import ls
+
+        exc = TypeError
+        if IS_PY2 and MINOR_VER == 6:
+            exc = ValueError
+
+        self.assertRaises(exc, ls, _env="XXX")
+        self.assertRaises(exc, ls, _env={"foo": 123})
+        self.assertRaises(exc, ls, _env={123: "bar"})
+
     def test_exception(self):
         from sh import ErrorReturnCode_2
 
@@ -595,7 +679,7 @@ sys.stdout.write("line2\\n")
 exit(2)
 """)
 
-        py2 = create_tmp_test("") 
+        py2 = create_tmp_test("")
 
         def fn():
             list(python(python(py.name, _piped=True), "-u", py2.name, _iter=True))
@@ -612,7 +696,7 @@ sys.stdout.write("line2\\n")
 exit(2)
 """)
 
-        py2 = create_tmp_test("") 
+        py2 = create_tmp_test("")
 
         def fn():
             python(python(py.name, _piped=True), "-u", py2.name)
@@ -642,6 +726,7 @@ exit(2)
 
         self.assertEqual(Command(which("ls")), ls)
 
+
     def test_doesnt_execute_directories(self):
         save_path = os.environ['PATH']
         bin_dir1 = tempfile.mkdtemp()
@@ -827,11 +912,19 @@ print(sys.argv[1])
                 _long_prefix="-custom-").strip()
         self.assertEqual(out, "-custom-long-option=underscore")
 
+        out = python(py.name, {"long-option": True},
+                _long_prefix="-custom-").strip()
+        self.assertEqual(out, "-custom-long-option")
+
         # test baking too
         out = python.bake(py.name, {"long-option": "underscore"},
                 _long_prefix="-baked-")().strip()
         self.assertEqual(out, "-baked-long-option=underscore")
 
+        out = python.bake(py.name, {"long-option": True},
+                _long_prefix="-baked-")().strip()
+        self.assertEqual(out, "-baked-long-option")
+
 
     def test_command_wrapper(self):
         from sh import Command, which
@@ -863,10 +956,10 @@ print(sys.argv[1])
 
     def test_background_exception(self):
         from sh import ls, ErrorReturnCode_1, ErrorReturnCode_2
-        p = ls("/ofawjeofj", _bg=True) # should not raise
+        p = ls("/ofawjeofj", _bg=True, _bg_exc=False) # should not raise
 
         exc_to_test = ErrorReturnCode_2
-        if IS_OSX: exc_to_test = ErrorReturnCode_1
+        if IS_MACOS: exc_to_test = ErrorReturnCode_1
         self.assertRaises(exc_to_test, p.wait) # should raise
 
 
@@ -1321,7 +1414,7 @@ import sys
 import os
 import time
 
-for i in range(5): 
+for i in range(5):
     print(i)
     time.sleep(.5)
 """)
@@ -1357,7 +1450,7 @@ import sys
 import os
 import time
 
-for i in range(5): 
+for i in range(5):
     print(i)
     time.sleep(.5)
 """)
@@ -1396,7 +1489,7 @@ import signal
 def sig_handler(sig, frame):
     print(10)
     exit(0)
-    
+
 signal.signal(signal.SIGINT, sig_handler)
 
 for i in range(5):
@@ -1426,7 +1519,7 @@ import sys
 import os
 import time
 
-for i in range(42): 
+for i in range(42):
     print(i)
     sys.stdout.flush()
 """)
@@ -1488,7 +1581,7 @@ sys.stderr.write("stderr")
 import sys
 import os
 
-for i in range(42): 
+for i in range(42):
     sys.stderr.write(str(i)+"\\n")
 """)
 
@@ -1597,7 +1690,7 @@ import sys
 import os
 
 for i in range(42):
-    sys.stderr.write(str(i * 2)+"\\n") 
+    sys.stderr.write(str(i * 2)+"\\n")
     print(i)
 """)
 
@@ -1634,10 +1727,10 @@ sys.stdout.write(sys.argv[1])
 
     def test_fg(self):
         py = create_tmp_test("exit(0)")
-        # notice we're using `sh.python`, and not `python`.  this is because
+        # notice we're using `system_python`, and not `python`.  this is because
         # `python` has an env baked into it, and we want `_env` to be None for
         # coverage
-        sh.python(py.name, _fg=True)
+        system_python(py.name, _fg=True)
 
     def test_fg_env(self):
         py = create_tmp_test("""
@@ -1682,6 +1775,26 @@ exit(49)
         self.assertEqual(str(pwd(_cwd="/etc")), realpath("/etc") + "\n")
 
 
+    def test_cwd_fg(self):
+        td = realpath(tempfile.mkdtemp())
+        py = create_tmp_test("""
+import sh
+import os
+from os.path import realpath
+orig = realpath(os.getcwd())
+print(orig)
+sh.pwd(_cwd="{newdir}", _fg=True)
+print(realpath(os.getcwd()))
+""".format(newdir=td))
+        
+        orig, newdir, restored = python(py.name).strip().split("\n")
+        newdir = realpath(newdir)
+        self.assertEqual(newdir, td)
+        self.assertEqual(orig, restored)
+        self.assertNotEqual(orig, newdir)
+        os.rmdir(td)
+
+
     def test_huge_piped_data(self):
         from sh import tr
 
@@ -1876,8 +1989,8 @@ sys.stdout.write("line1")
         started = time()
         try:
             sh.sleep(sleep_for, _timeout=timeout).wait()
-        except sh.TimeoutException:
-            pass
+        except sh.TimeoutException as e:
+            self.assertEqual(e.full_cmd, '/bin/sleep 3')
         else:
             self.fail("no timeout exception")
         elapsed = time() - started
@@ -1891,6 +2004,24 @@ sys.stdout.write("line1")
         self.assertTrue(abs(elapsed - 1) < 0.5)
 
 
+    def test_timeout_wait(self):
+        started = time.time()
+        p = sh.sleep(3, _bg=True)
+        self.assertRaises(sh.TimeoutException, p.wait, timeout=1)
+
+
+    def test_timeout_wait_overstep(self):
+        started = time.time()
+        p = sh.sleep(1, _bg=True)
+        p.wait(timeout=5)
+
+
+    def test_timeout_wait_negative(self):
+        started = time.time()
+        p = sh.sleep(3, _bg=True)
+        self.assertRaises(RuntimeError, p.wait, timeout=-3)
+
+
     def test_binary_pipe(self):
         binary = b'\xec;\xedr\xdbF'
 
@@ -2058,8 +2189,10 @@ time.sleep(3)
 """)
 
         parent = create_tmp_test("""
+import sys
 import sh
-p = sh.python("{child_file}", _bg=True, _new_session=False)
+python = sh.Command(sys.executable)
+p = python("{child_file}", _bg=True, _new_session=False)
 print(p.pid)
 print(p.process.pgid)
 p.wait()
@@ -2162,10 +2295,9 @@ p.wait()
         from sh import ls
 
         # sanity check
-        non_exist_dir = join(tempdir, "aowjgoahewro") 
+        non_exist_dir = join(tempdir, "aowjgoahewro")
         self.assertFalse(exists(non_exist_dir))
-
-        self.assertRaises(OSError, ls, _cwd=non_exist_dir)
+        self.assertRaises(sh.ForkException, ls, _cwd=non_exist_dir)
 
 
     # https://github.com/amoffat/sh/issues/176
@@ -2525,7 +2657,8 @@ import sys
 child_file = sys.argv[1]
 output_file = sys.argv[2]
 
-os.spawnlp(os.P_NOWAIT, "python", "python", child_file, output_file)
+python_name = os.path.basename(sys.executable)
+os.spawnlp(os.P_NOWAIT, python_name, python_name, child_file, output_file)
 time.sleep(1) # give child a chance to set up
 """)
 
@@ -2618,6 +2751,27 @@ class MockTests(BaseTests):
 
 
 class MiscTests(BaseTests):
+    def test_pickling(self):
+        import pickle
+
+        py = create_tmp_test("""
+import sys
+sys.stdout.write("some output")
+sys.stderr.write("some error")
+exit(1)
+""")
+
+        try:
+            python(py.name)
+        except sh.ErrorReturnCode as e:
+            restored = pickle.loads(pickle.dumps(e))
+            self.assertEqual(restored.stdout, b"some output")
+            self.assertEqual(restored.stderr, b"some error")
+            self.assertEqual(restored.exit_code, 1)
+        else:
+            self.fail("Didn't get an exception")
+
+
     @requires_poller("poll")
     def test_fd_over_1024(self):
         py = create_tmp_test("""print("hi world")""")
@@ -2640,13 +2794,6 @@ class MiscTests(BaseTests):
     def test_args_deprecated(self):
         self.assertRaises(DeprecationWarning, sh.args, _env={})
 
-    def test_cant_import_all(self):
-        def go():
-            # we have to use exec, because in py3, this syntax raises a
-            # SyntaxError upon compilation
-            exec("from sh import *")
-        self.assertRaises(RuntimeError, go)
-
     def test_percent_doesnt_fail_logging(self):
         """ test that a command name doesn't interfere with string formatting in
         the internal loggers """
@@ -2657,17 +2804,16 @@ print("cool")
         out = python(py.name, "%%")
         out = python(py.name, "%%%")
 
-
     # TODO
     # for some reason, i can't get a good stable baseline measured in this test
     # on osx.  so skip it for now if osx
-    @not_osx
+    @not_macos
     @requires_progs("lsof")
     def test_no_fd_leak(self):
         import sh
         import os
         from itertools import product
-        
+
         # options whose combinations can possibly cause fd leaks
         kwargs = {
             "_tty_out": (True, False),
@@ -2769,10 +2915,11 @@ print("hi")
     def test_unicode_path(self):
         from sh import Command
 
-        py = create_tmp_test("""#!/usr/bin/env python
+        python_name = os.path.basename(sys.executable)
+        py = create_tmp_test("""#!/usr/bin/env {0}
 # -*- coding: utf8 -*-
 print("字")
-""", "字", delete=False)
+""".format(python_name), prefix="字", delete=False)
 
         try:
             py.close()
@@ -2875,11 +3022,11 @@ sys.stdout.write(repr(res))
         def f1():
             with p1:
                 time.sleep(1)
-                results[0] = str(sh.python("one"))
+                results[0] = str(system_python("one"))
 
         def f2():
             with p2:
-                results[1] = str(sh.python("two"))
+                results[1] = str(system_python("two"))
 
         t1 = threading.Thread(target=f1)
         t1.start()
@@ -3007,7 +3154,7 @@ class ExecutionContextTests(unittest.TestCase):
         import sh
         _sh = sh()
         omg = _sh
-        from omg import python
+        from omg import cat
 
     def test_importer_only_works_with_sh(self):
         def unallowed_import():
@@ -3015,6 +3162,31 @@ class ExecutionContextTests(unittest.TestCase):
             from _os import path
         self.assertRaises(ImportError, unallowed_import)
 
+    def test_reimport_from_cli(self):
+        # The REPL and CLI both need special handling to create an execution context that is safe to
+        # reimport
+        if IS_PY3:
+            cmdstr = '; '.join(('import sh, io, sys',
+                                'out = io.StringIO()',
+                                '_sh = sh(_out=out)',
+                                'import _sh',
+                                '_sh.echo("-n", "TEST")',
+                                'sys.stderr.write(out.getvalue())',
+                                ))
+        else:
+            cmdstr = '; '.join(('import sh, StringIO, sys',
+                                'out = StringIO.StringIO()',
+                                '_sh = sh(_out=out)',
+                                'import _sh',
+                                '_sh.echo("-n", "TEST")',
+                                'sys.stderr.write(out.getvalue())',
+                                ))
+
+        err = StringIO()
+
+        python('-c', cmdstr, _err=err)
+        self.assertEqual('TEST', err.getvalue())
+
 
 if __name__ == "__main__":
     root = logging.getLogger()