Repository: psychopy/psychopy

Issue:
GUI install under OS X: handling of framework-python
TL;DR If you want to run the `psychopyApp.py` GUI on an OS X dev-install, call

```bash
python.app `which psychopyApp.py`  # or equivalently use: pythonw
```

## Description

Even on successful OS X compilations, and after this locale-fix #1325, the distutils install leaves the script `psychopyApp.py` with the shebang

```bash
#!/Users/cjayb/miniconda_envs/psychopy/bin/python
```

_i.e._, one that is not compiled against the "Framework" libs and therefore does not get access to the screen:

```bash
This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.
```

Both `pythonw` and `python.app` are [easily installed using conda](https://gist.github.com/cjayb/67d8f6ada85f8fcd58ff1d4e1fc20764), and it's possible to specify the interpreter for `scripts`:

```bash
python setup.py build --executable=`which python.app`
```

I've also tried simply:

```bash
python.app setup.py install --user
```

but I can't get the pieces to work together! Whatever I try, the shebang remains just `python`.

Any ideas on whether this can be fixed in `setup.py`?
GUI install under OS X: handling of framework-python
TL;DR If you want to run the `psychopyApp.py` GUI on an OS X dev-install, call

```bash
python.app `which psychopyApp.py`  # or equivalently use: pythonw
```

## Description

Even on successful OS X compilations, and after this locale-fix #1325, the distutils install leaves the script `psychopyApp.py` with the shebang

```bash
#!/Users/cjayb/miniconda_envs/psychopy/bin/python
```

_i.e._, one that is not compiled against the "Framework" libs and therefore does not get access to the screen:

```bash
This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.
```

Both `pythonw` and `python.app` are [easily installed using conda](https://gist.github.com/cjayb/67d8f6ada85f8fcd58ff1d4e1fc20764), and it's possible to specify the interpreter for `scripts`:

```bash
python setup.py build --executable=`which python.app`
```

I've also tried simply:

```bash
python.app setup.py install --user
```

but I can't get the pieces to work together! Whatever I try, the shebang remains just `python`.

Any ideas on whether this can be fixed in `setup.py`?


Discussion:
I don't really understand the aim here. You're trying to construct the app bundle from a conda installation? Or you're tryng to *avoid* a conda python getting involved?
To be fair though I don't know the answer as I use regular python.org install and pip libs.
This is just a minor inconvenience, and a consequence of using the conda-packaged python over the vanilla .org, which compiles against OS X frameworks and thus doesn't suffer from `wx` lacking access to the screen. No need to complicate `setup.py` for this!

Feel free to close this. If you like, you can add the very first line of code to your dev-docs, in case conda-users find it useful:

```bash
python.app `which psychopyApp.py`
```
Anaconda is very common among scientists and I recommend it to all my colleagues and students. I can reproduce @cjayb's report:

```zsh
[hoechenberger:~]$ source activate new_psychopy
(new_psychopy) [hoechenberger:~]$ which python
/Users/hoechenberger/anaconda3/envs/new_psychopy/bin/python
(new_psychopy) [hoechenberger:~]$ python --version
Python 2.7.13 :: Anaconda 4.3.0 (x86_64)
(new_psychopy) [hoechenberger:~]$ python ~/Development/psychopy/psychopy/app/psychopyApp.py
This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.
(new_psychopy) [hoechenberger:~]$ pythonw ~/Development/psychopy/psychopy/app/psychopyApp.py
2017-03-22 13:14:07.133 python[6020:537744] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to /var/folders/75/zwcv5c8j5sz3c1_6m60xpct80000gp/T/com.continuum.python.savedState
^C
(new_psychopy) [hoechenberger:~]$
```
I think `setup.py` should check if the current platform is macOS ("Darwin") and use `pythonw` in that case.
By "use", I mean add `pythonw` to the shebang line instead of `python`.
I'm happy to see changes to setup.py if those will help (and assuming they won't break other things). I don't think I've ever made use of the shebang line. Always specify the python that will launch the app. My `.bash_profile` has this (and obviously pythonw would be easy to switch in)

`alias psp='python $HOME/code/psychopy/git/psychopy/app/psychopyApp.py'`

and then I launch the app with `psp`. Maybe that's why I haven't really understood what the issue is here. But, as I say, if there's something that would be useful and won't mess with anyone else then I'm happy for it to be done. (There have been a number of changes to the shebang line, always by other people)
I also wonder whether the line should not be "location-agnostic", i.e., `#!/usr/bin/env python(w)2` instead of specifying an exact path? Could that have any undesired side-effects in realistic situations?

x-ref to Anaconda issue tracker:
[pythonw should be default in OS-X ](https://github.com/ContinuumIO/anaconda-issues/issues/199)
On Unix-like platforms, couldn't we simply create a symlink to `psychopy/psychopy/app/psychopyApp.py`? (Which *does* have a shebang line) Or is this not supported by distutils? The target would have to have the executable bit set, though.
Ah no, scratch that. The symlink would require the installer to alter the original file `psychopy/app/psychopyApp.py` to use `pythonw` in the shebang line. Argh! So we should definitely stick to some sort of "separate invocation script", which the setup process would adjust such that it would run `psychopyApp.py` with the correct interpreter.
Thinking some more about this, I'm starting to believe this is actually a bug in Anaconda that we shouldn't have to deal with. I would propose to update the install documentation and wait for the Anaconda people to fix it already. There's many people affected by this odd behavior, not just users of PsychoPy...
...or setuptools should work around it:
https://github.com/pypa/setuptools/issues/410
Related to the comment 5 posts above about a "location-agnostic" path: currently, installing from pip (the wheel) on linux causes the first line on psychopyApp.py to be `#!/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python`. A location-agnostic path would presumably also fix that.
The problem is that setuptools automatically changes the shebang line:
https://docs.python.org/2/distutils/setupscript.html#installing-scripts
OK, in the repository the shebang is `#!/usr/bin/env python2`
This was being changed by setuptools during build of eggs and wheels to the path on my local machine (dumbest idea ever)
Eggs installed with easy_install actually then get changed *again* during install so those were fine but pip doesn't touch the shebang

BUT, this was using setuptools 28.3.0. I've just updated setuptools to 34.3.2 on my build machine and it seems not to touch shebnag anymore for script copying so we'll stick with `#!/usr/bin/env python2`

Hopefully that will fix for everyone. In the meantime you can:

- install the source install (zip file) instead of the egg/wheel
- install with easy_install
- avoid using the shebang line altogether by specifying your python executable when launching the script

Oh, by the way, I'm happy for the shebang to be changed to pythonw2 for the psyhopyApp.py if you think that will work everywhere
> Oh, by the way, I'm happy for the shebang to be changed to pythonw2 for the psyhopyApp.py if you think that will work everywhere

I'm afraid `pythonw` is not universally available on all platforms.
I don't really understand the aim here. You're trying to construct the app bundle from a conda installation? Or you're tryng to *avoid* a conda python getting involved?
To be fair though I don't know the answer as I use regular python.org install and pip libs.
This is just a minor inconvenience, and a consequence of using the conda-packaged python over the vanilla .org, which compiles against OS X frameworks and thus doesn't suffer from `wx` lacking access to the screen. No need to complicate `setup.py` for this!

Feel free to close this. If you like, you can add the very first line of code to your dev-docs, in case conda-users find it useful:

```bash
python.app `which psychopyApp.py`
```
Anaconda is very common among scientists and I recommend it to all my colleagues and students. I can reproduce @cjayb's report:

```zsh
[hoechenberger:~]$ source activate new_psychopy
(new_psychopy) [hoechenberger:~]$ which python
/Users/hoechenberger/anaconda3/envs/new_psychopy/bin/python
(new_psychopy) [hoechenberger:~]$ python --version
Python 2.7.13 :: Anaconda 4.3.0 (x86_64)
(new_psychopy) [hoechenberger:~]$ python ~/Development/psychopy/psychopy/app/psychopyApp.py
This program needs access to the screen.
Please run with a Framework build of python, and only when you are
logged in on the main display of your Mac.
(new_psychopy) [hoechenberger:~]$ pythonw ~/Development/psychopy/psychopy/app/psychopyApp.py
2017-03-22 13:14:07.133 python[6020:537744] ApplePersistenceIgnoreState: Existing state will not be touched. New state will be written to /var/folders/75/zwcv5c8j5sz3c1_6m60xpct80000gp/T/com.continuum.python.savedState
^C
(new_psychopy) [hoechenberger:~]$
```
I think `setup.py` should check if the current platform is macOS ("Darwin") and use `pythonw` in that case.
By "use", I mean add `pythonw` to the shebang line instead of `python`.
I'm happy to see changes to setup.py if those will help (and assuming they won't break other things). I don't think I've ever made use of the shebang line. Always specify the python that will launch the app. My `.bash_profile` has this (and obviously pythonw would be easy to switch in)

`alias psp='python $HOME/code/psychopy/git/psychopy/app/psychopyApp.py'`

and then I launch the app with `psp`. Maybe that's why I haven't really understood what the issue is here. But, as I say, if there's something that would be useful and won't mess with anyone else then I'm happy for it to be done. (There have been a number of changes to the shebang line, always by other people)
I also wonder whether the line should not be "location-agnostic", i.e., `#!/usr/bin/env python(w)2` instead of specifying an exact path? Could that have any undesired side-effects in realistic situations?

x-ref to Anaconda issue tracker:
[pythonw should be default in OS-X ](https://github.com/ContinuumIO/anaconda-issues/issues/199)
On Unix-like platforms, couldn't we simply create a symlink to `psychopy/psychopy/app/psychopyApp.py`? (Which *does* have a shebang line) Or is this not supported by distutils? The target would have to have the executable bit set, though.
Ah no, scratch that. The symlink would require the installer to alter the original file `psychopy/app/psychopyApp.py` to use `pythonw` in the shebang line. Argh! So we should definitely stick to some sort of "separate invocation script", which the setup process would adjust such that it would run `psychopyApp.py` with the correct interpreter.
Thinking some more about this, I'm starting to believe this is actually a bug in Anaconda that we shouldn't have to deal with. I would propose to update the install documentation and wait for the Anaconda people to fix it already. There's many people affected by this odd behavior, not just users of PsychoPy...
...or setuptools should work around it:
https://github.com/pypa/setuptools/issues/410
Related to the comment 5 posts above about a "location-agnostic" path: currently, installing from pip (the wheel) on linux causes the first line on psychopyApp.py to be `#!/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python`. A location-agnostic path would presumably also fix that.
The problem is that setuptools automatically changes the shebang line:
https://docs.python.org/2/distutils/setupscript.html#installing-scripts
OK, in the repository the shebang is `#!/usr/bin/env python2`
This was being changed by setuptools during build of eggs and wheels to the path on my local machine (dumbest idea ever)
Eggs installed with easy_install actually then get changed *again* during install so those were fine but pip doesn't touch the shebang

BUT, this was using setuptools 28.3.0. I've just updated setuptools to 34.3.2 on my build machine and it seems not to touch shebnag anymore for script copying so we'll stick with `#!/usr/bin/env python2`

Hopefully that will fix for everyone. In the meantime you can:

- install the source install (zip file) instead of the egg/wheel
- install with easy_install
- avoid using the shebang line altogether by specifying your python executable when launching the script

Oh, by the way, I'm happy for the shebang to be changed to pythonw2 for the psyhopyApp.py if you think that will work everywhere
> Oh, by the way, I'm happy for the shebang to be changed to pythonw2 for the psyhopyApp.py if you think that will work everywhere

I'm afraid `pythonw` is not universally available on all platforms.

Resolving patch:
diff --git a/psychopy/app/psychopyApp.py b/psychopy/app/psychopyApp.py
--- a/psychopy/app/psychopyApp.py
+++ b/psychopy/app/psychopyApp.py
@@ -8,15 +8,27 @@
 from __future__ import absolute_import, print_function
 
 import sys
-from psychopy.app._psychopyApp import PsychoPyApp, __version__
 
 # fix macOS locale-bug on startup: sets locale to LC_ALL (must be defined!)
 import psychopy.locale_setup  # noqa
 
+
 # NB the PsychoPyApp classes moved to _psychopyApp.py as of version 1.78.00
 # to allow for better upgrading possibilities from the mac app bundle. this
 # file now used solely as a launcher for the app, not as the app itself.
 
+
+def start_app():
+    from psychopy.app._psychopyApp import PsychoPyApp
+
+    showSplash = True
+    if '--no-splash' in sys.argv:
+        showSplash = False
+        del sys.argv[sys.argv.index('--no-splash')]
+    app = PsychoPyApp(0, showSplash=showSplash)
+    app.MainLoop()
+
+
 if __name__ == '__main__':
     if '-x' in sys.argv:
         # run a .py script from the command line using StandAlone python
@@ -26,8 +38,10 @@
         core.shellCall([sys.executable, os.path.abspath(targetScript)])
         sys.exit()
     if '-v' in sys.argv or '--version' in sys.argv:
-        info = 'PsychoPy2, version %s (c)Jonathan Peirce 2015, GNU GPL license'
-        print(info % __version__)
+        from psychopy import __version__
+        msg = ('PsychoPy2, version %s (c)Jonathan Peirce 2015, GNU GPL license'
+               % __version__)
+        print(msg)
         sys.exit()
     if '-h' in sys.argv or '--help' in sys.argv:
         print("""Starts the PsychoPy2 application.
@@ -56,10 +70,27 @@
 """)
         sys.exit()
 
+    if (sys.platform == 'darwin' and
+            ('| packaged by conda-forge |' in sys.version or
+             '|Anaconda' in sys.version)):
+
+        # On macOS with Anaconda, GUI applications need to be run using
+        # `pythonw`. Since we have no way to determine whether this is currently
+        # the case, we run this script again -- ensuring we're definitely using
+        # pythonw.
+        import os
+        env = os.environ
+        PYTHONW = env.get('PYTHONW', 'False')
+
+        if PYTHONW != 'True':
+            from psychopy import core
+            cmd = [sys.executable + 'w', __file__]
+            if '--no-splash' in sys.argv:
+                cmd.append('--no-splash')
+
+            core.shellCall(cmd, env=dict(env, PYTHONW='True'))
+            sys.exit()
+        else:
+            start_app()
     else:
-        showSplash = True
-        if '--no-splash' in sys.argv:
-            showSplash = False
-            del sys.argv[sys.argv.index('--no-splash')]
-        app = PsychoPyApp(0, showSplash=showSplash)
-        app.MainLoop()
+        start_app()
diff --git a/psychopy/core.py b/psychopy/core.py
--- a/psychopy/core.py
+++ b/psychopy/core.py
@@ -75,12 +75,46 @@ def quit():
     sys.exit(0)  # quits the python session entirely
 
 
-def shellCall(shellCmd, stdin='', stderr=False):
+def shellCall(shellCmd, stdin='', stderr=False, env=None, encoding='utf-8'):
     """Call a single system command with arguments, return its stdout.
     Returns stdout, stderr if stderr is True.
     Handles simple pipes, passing stdin to shellCmd (pipes are untested
     on windows) can accept string or list as the first argument
+
+    Parameters
+    ----------
+    shellCmd : str, or iterable
+        The command to execute, and its respective arguments.
+
+    stdin : str, or None
+        Input to pass to the command.
+
+    stderr : bool
+        Whether to return the standard error output once execution is finished.
+
+    env : dict
+        The environment variables to set during execution.
+
+    encoding : str
+        The encoding to use for communication with the executed command.
+        This argument will be ignored on Python 2.7.
+
+    Notes
+    -----
+    We use ``subprocess.Popen`` to execute the command and establish
+    `stdin` and `stdout` pipes.
+    Python 2.7 always opens the pipes in text mode; however,
+    Python 3 defaults to binary mode, unless an encoding is specified.
+    To unify pipe communication across Python 2 and 3, we now provide an
+    `encoding` parameter, enforcing `utf-8` text mode by default.
+    This parameter is present from Python 3.6 onwards; using an older
+    Python 3 version will raise an exception. The parameter will be ignored
+    when running Python 2.7.
+
     """
+    if env is None:
+        env = dict()
+
     if type(shellCmd) == str:
         # safely split into cmd+list-of-args, no pipes here
         shellCmdList = shlex.split(shellCmd)
@@ -90,16 +124,33 @@ def shellCall(shellCmd, stdin='', stderr=False):
     elif type(shellCmd) in (list, tuple):  # handles whitespace in filenames
         shellCmdList = shellCmd
     else:
-        return None, 'shellCmd requires a list or string'
+        msg = 'shellCmd requires a string or iterable.'
+        raise TypeError(msg)
+
     bytesObjects = []
     for obj in shellCmdList:
         if type(obj) != bytes:
             bytesObjects.append(obj.encode('utf-8'))
         else:
             bytesObjects.append(obj)
-    proc = subprocess.Popen(bytesObjects, stdin=subprocess.PIPE,
-                            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
+    # Since Python 3.6, we can use the `encoding` parameter.
+    if PY3:
+        if sys.version_info.minor >= 6:
+            proc = subprocess.Popen(bytesObjects, stdin=subprocess.PIPE,
+                                    stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    encoding=encoding, env=env)
+        else:
+            msg = 'shellCall() requires Python 2.7, or 3.6 and newer.'
+            raise RuntimeError(msg)
+    else:
+        proc = subprocess.Popen(bytesObjects, stdin=subprocess.PIPE,
+                                stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE, env=env)
+
     stdoutData, stderrData = proc.communicate(stdin)
+
     del proc
     if stderr:
         return stdoutData.strip(), stderrData.strip()


Test patch:
diff --git a/psychopy/tests/test_misc/test_core.py b/psychopy/tests/test_misc/test_core.py
--- a/psychopy/tests/test_misc/test_core.py
+++ b/psychopy/tests/test_misc/test_core.py
@@ -396,22 +396,59 @@ def test_quit():
 
 
 @pytest.mark.shellCall
-def test_shellCall():
-    if PY3:
-        # This call to shellCall from tests is failing from Python3
-        # but maybe it just isn't a great test anyway?!
-        # shellCall is used by PsychoPy Tools>Run and works on Py3 there!
-        pytest.xfail(reason="Failing on Py3")
-    msg = 'echo'
-    cmd = ('grep', 'findstr')[sys.platform == 'win32']
-
-    for arg1 in [[cmd, msg],  cmd+' '+msg]:
-        echo = shellCall(arg1, stdin=msg)
-        assert echo == msg
-        echo, se = shellCall(arg1, stdin=msg, stderr=True)
-        assert echo == msg
-    echo, se = shellCall(12, stdin=msg)
-    assert echo is None
+class Test_shellCall(object):
+    def setup_class(self):
+        if sys.platform == 'win32':
+            self.cmd = 'findstr'
+            self.env_cmd = 'set'
+        else:
+            self.cmd = 'grep'
+            self.env_cmd = 'env'
+
+        self.msg = 'echo'
+
+    def test_invalid_argument(self):
+        with pytest.raises(TypeError):
+            shellCall(12345)
+
+    def test_stdin(self):
+        echo = shellCall([self.cmd, self.msg], stdin=self.msg)
+        assert echo == self.msg
+
+        echo = shellCall(self.cmd + ' ' + self.msg, stdin=self.msg)
+        assert echo == self.msg
+
+    def test_stderr(self):
+        _, se = shellCall([self.cmd, self.msg], stderr=True)
+        assert se == ''
+
+        _, se = shellCall(self.cmd + ' ' + self.msg, stderr=True)
+        assert se == ''
+
+    def test_stdin_and_stderr(self):
+        echo, se = shellCall([self.cmd, self.msg], stdin='echo', stderr=True)
+        assert echo == self.msg
+        assert se == ''
+
+        echo, se = shellCall(self.cmd + ' ' + self.msg, stdin='echo',
+                             stderr=True)
+        assert echo == self.msg
+        assert se == ''
+
+    def test_encoding(self):
+        shellCall([self.cmd, self.msg], stdin=self.msg, encoding='utf-8')
+
+        if PY3:
+            with pytest.raises(TypeError):
+                shellCall([self.cmd, self.msg], stdin=self.msg, encoding=None)
+
+    def test_env(self):
+        echo = shellCall(self.env_cmd)
+        assert echo == ''
+
+        echo = shellCall(self.env_cmd, env=dict(FOO='1'))
+        assert echo == 'FOO=1'
+
 
 
 if __name__ == '__main__':