Repository: psychopy/psychopy

Issue:
Syntax error on shebang line in PsychoPy (Python 3) when compiling code from Builder
When compiling Python code from Builder using PsychoPy running on Python 3 (on Windows 10), the shebang line contains a syntax error:
```
ï»¿#!/usr/bin/env python
# -*- coding: utf-8 -*-
```


Discussion:
<strike>Could you please attach a file like this for inspection (simply drag & drop it into the comment field)?</strike> This could be the BOM we're adding now by default in many places – don't remember if we modified Builder output too? (not intentionally, I would guess!)

When you say "contains a syntax error", do you mean this is how it gets displayed in Coder after compiling? Do you see this symbol in a "proper" editor too? (I suspect not) – Could mean we'd need to adjust the file-loading mechanism in Coder (to use `utf-8-sig` encoding)
So yeah I'm very certain it's a BOM, see https://en.wikipedia.org/wiki/Byte_order_mark#Byte_order_marks_by_encoding ("Bytes as CP1252 characters" column for UTF-8, i.e., first row – that's your sequence.)

So – Coder seems to open the file using the legacy CP1252 encoding when it should be using `utf-8-sig`, I guess.
Daaamn, this seems to be a bit more complicated. As I understand it, we _already_ use the correct encoding when _opening_ the file:
https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/app/coder/coder.py#L2181

Could it be that it's merely a problem with _displaying_ the code? Can you _run_ the file?
I suspect we should make use of wxPython's [`StyledTextCtrl.StyleSetFontEncoding()`](https://wxpython.org/Phoenix/docs/html/wx.stc.StyledTextCtrl.html#wx.stc.StyledTextCtrl.StyleSetFontEncoding) for the respective Coder widget that displays the code.
Thanks Richard, I cannot run the file from Coder, it throws the ```Syntax Error: invalid character in identifier```, however Builder itself outputs the code correctly, so I guess we see why the outputs two differ. What I mean is, the exp_lastrun.py script is ok.




@hoechenberger maybe it's just csv files that need the BOM adding?
Yes, I think it is this causing it - removing the sig fixes the problem:        

https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/scripts/psyexpCompile.py#L149
> What I mean is, the exp_lastrun.py script is ok.

Was already very surprised since at first I read your comment as if this was not the case. For reference, here's a small test case:
```python
import pathlib


def create_file(*, filename, content):
    with open(filename, 'w', encoding='utf-8-sig') as f:
        f.write(content)


filename = pathlib.Path('/tmp/foo.py')
content = """#!/usr/bin/env python
# -*- coding: utf-8 -*-

print('Hello there.')
"""

create_file(filename=filename, content=content)
```

The resulting file has a BOM and can be run without any issues.
```
(base) [hoechenberger:/tmp]$ python foo.py
Hello there.
(base) [hoechenberger:/tmp]$
```

> @hoechenberger maybe it's just csv files that need the BOM adding?

Actually, as far as I can see, no: if we want to support e.g. Chinese string literals in the scripts, they need to be saved as `utf-8-sig` as well.

Since `exp_lastrun.py` seems to be alright – which should contain the exact same code as what is displayed in the Coder, right? –, I think we should try setting the encoding for the `StyleTextCtrl` (as proposed here: https://github.com/psychopy/psychopy/issues/2235#issuecomment-458229997), since by default WX uses the system default encoding, which, on your UK or US Windows system, would typically be non-UTF-8, meaning the BOM won't be handled correctly.
@dvbridges If you open `exp_lastrun.py` in Coder, do you see the same issue?
No, the ```exp_lastrun.py``` has no issues in Coder - so probably specific to how ```psyexpCompile``` compiles the script.
Both [`BuilderFrame.runFile()`](https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/app/builder/builder.py#L2125) (Run Button) and [`BuilderFrame.compileScript`](https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/app/builder/builder.py#L2311) (Compile Button) call [`BuilderFrame.generateScript()`](https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/app/builder/builder.py#L2316), which invokes `psychopy.scripts.psyexpCompile()`. Should produce the exact same output!
> I think we should try setting the encoding for the `StyleTextCtrl` (as proposed here: [#2235 (comment)](https://github.com/psychopy/psychopy/issues/2235#issuecomment-458229997)), since by default WX uses the system default encoding, which, on your UK or US Windows system, would typically be non-UTF-8, meaning the BOM won't be handled correctly.

Then again, we open and read the file with the correct encoding before passing the contents to WX, so the BOM should actually have been removed by then, no?
Yes, ok how about this line - as it is specific to Python 3, and this is where the file is read after being compiled - open the file with utf-8-sig encoding

https://github.com/psychopy/psychopy/blob/64449dd68b7e5c0d01d9f893c333821edebe1289/psychopy/app/coder/coder.py#L2138


Yes, cool, that could be it! Could you try removing the entire block:
```python
            if PY3:
                doc.SetText(open(filename).read())
            else:
                doc.SetText(open(filename).read().decode('utf8'))
```
and replace it with:
```python
            with io.open(filename, 'r', encoding='utf-8-sig') as f:
                doc.SetText(f.read())
```
And see if that fixes the issue?
Yes, it works :). Good idea using the same the context manager.
Great! Can you submit a PR? I must have overlooked these lines during the switch to `utf-8-sig`. Wonder if there are more bugs of this kind!
Yep, no problem

Resolving patch:
diff --git a/psychopy/app/coder/coder.py b/psychopy/app/coder/coder.py
--- a/psychopy/app/coder/coder.py
+++ b/psychopy/app/coder/coder.py
@@ -2134,10 +2134,8 @@ def fileReload(self, event, filename=None, checkSave=False):
 
         # is the file still there
         if os.path.isfile(filename):
-            if PY3:
-                doc.SetText(open(filename).read())
-            else:
-                doc.SetText(open(filename).read().decode('utf8'))
+            with io.open(filename, 'r', encoding='utf-8-sig') as f:
+                doc.SetText(f.read())
             doc.fileModTime = os.path.getmtime(filename)
             doc.EmptyUndoBuffer()
             doc.Colourise(0, -1)


Test patch: