Repository: bokeh/bokeh

Issue:
Trouble with encoding on Windows
From SO: http://stackoverflow.com/questions/24075840/python-trouble-with-encoding-on-windows-bokeh-plotting-library

Proposed solution, add "encoding" parameter defaulting to UTF-8.



Discussion:
Proposed solutions seems good.

As requested on an original SO answer (http://stackoverflow.com/a/24090474/1414455), my architecture is 64-bit Windows 7 SP1.

Python is 64-bit Python 3.3.0. Latest version of Bokeh installed from Github yesterday.

Can you also provide the locale you are using? Thanks

A call to locale.getdefaultlocale() returns ('en_US', 'cp1252').

Pretty sure my vm was set to that too. I definitely don't understand this. Were there any special characters in your script?

> On Jun 7, 2014, at 16:49, fgnu notifications@github.com wrote:
> 
> A call to locale.getdefaultlocale() returns ('en_US', 'cp1252').
> 
> —
> Reply to this email directly or view it on GitHub.

@fgnu any update?

absent more info, closing. Please re-open if more information comes to light. 

I just had this issue. I am on 64-bit Windows 7 SP1, with 64 BIt Python 3.4. If I change the lines in plotting.py to     

``` python
with open(filename, "wb") as f:
        f.write(html.encode('utf-8'))
```

It works.

I will submit a pull request.

From SO: https://stackoverflow.com/questions/46187520/python-trouble-with-encoding-on-windows-when-run-bokeh-serve-bokeh-plotting-li

I think there is another missing of using 'utf-8' [here](https://github.com/bokeh/bokeh/blob/f3d13963a36130ec126f0a1e87d3abcf8e5870f7/bokeh/application/handlers/script.py#L53). If everyone feels ok, I can pull a request. The current status is if you are under a Windows system, your script file contains some special charactor, `bokeh serve` won't work no matter your script file is saved in utf-8 format or some codec like 'gbk'.

I have to chime in on this because I encountered the same bug as pointed out previously by @gepcel 
Extending that particular [line](https://github.com/bokeh/bokeh/blob/c8982366e3676757c749c2c2b66ea46a737ba7e9/bokeh/application/handlers/script.py#L84) with ```encoding='utf8'``` helps to make my app run. Without that, python switches to cp1252 which leads to a crash.

Tested on Windows 7, German locale, Python 3.6.5 & bokeh 0.13.0

Happy to consider a PR if someone wants to submit one. I don't really have the means to test that this works in the situations people are reporting, only to test that it does not break in current situations. 
Looks like the solution suggested here will fix the issue with python3, but break things with python2. There was an unsuccessful trying, see discussion #7305.

As for now, my solution for myself is manually changing this line of code everytime I installed or updated bokeh.
I made some tests with this script:
```python
# -*- coding: utf-8 -*-
from bokeh.plotting import figure
from bokeh.io import curdoc
fig = figure(title="Entête", plot_width=400, plot_height=200)
curdoc().add_root(fig)
```
on 4 platforms:

- Windows 7 / python 2.7 / bokeh 0.13
  No issue (accented text properly rendered). Just this warning:
  ```
  M:\Anaconda3\envs\py27\lib\site-packages\bokeh\core\property\bases.py:197: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  return new == old
  ```

- Linux / python 2.7 / bokeh 0.13
  No issue (accented text properly rendered). Same warning.

- Windows 7 / python 3.6 / bokeh 0.13
  Accented characters are not properly rendered. First line `# -*- coding: utf-8 -*-`
  is not taken into account unlike python 2.7. With python 3.6 the script file is
  read according to `sys.getdefaultencoding()` (= cp1252 on my laptop configuration)

- Linux / python 3.6 / bokeh 0.13
  No issue (accented text properly rendered)

Instead of using `io.open` that breaks things python 2 version - cf. [#7305](https://github.com/bokeh/bokeh/pull/7305) -, maybe we could fix this issue by testing python version in script.py like this:

```python
if sys.version_info[0] > 2:
    with open(filename, 'r', encoding='utf-8') as f:
        kwargs['source'] = f.read()
else:
    with open(filename, 'r') as f:
        kwargs['source'] = f.read()
```

What's your opinion?
@lgasnier that might be reasonable. Can you push a PR so we can first see that all existing tests pass and also experiment with the change easily?
Tho, please use `if six.PY2` etc instead of checking `sys.version_info` manually.

Resolving patch:
diff --git a/bokeh/application/handlers/script.py b/bokeh/application/handlers/script.py
--- a/bokeh/application/handlers/script.py
+++ b/bokeh/application/handlers/script.py
@@ -49,6 +49,7 @@
 # Standard library imports
 
 # External imports
+import six
 
 # Bokeh imports
 from .code import CodeHandler
@@ -81,8 +82,19 @@ def __init__(self, *args, **kwargs):
             raise ValueError('Must pass a filename to ScriptHandler')
         filename = kwargs['filename']
 
-        with open(filename, 'r') as f:
-            kwargs['source'] = f.read()
+        # For Python 3, encoding must be set to utf-8 because:
+        # - when specifying an encoding in `io.open`, it doesn't
+        #   work with Python 2,
+        # - default encoding used by Python 3 `open` statement
+        #   is not 'utf-8' on Windows platform,
+        # - Python 3 `open` ignores le `coding` comment line.
+        # See https://github.com/bokeh/bokeh/pull/8202 for details.
+        if six.PY3:
+            with open(filename, 'r', encoding='utf-8') as f:
+                kwargs['source'] = f.read()
+        else:
+            with open(filename, 'r') as f:
+                kwargs['source'] = f.read()
 
         super(ScriptHandler, self).__init__(*args, **kwargs)
 


Test patch: