Repository: kivy/kivy

Issue:
Loading UTF8 KV file with Japanese (multi-byte?) text crashes Kivy under Windows
### Versions

* Python: 3.4.4
* OS: Windows 10 Home
* Kivy: 1.9.1
* Kivy installation method: followed https://kivy.org/docs/installation/installation-windows.html#installation

### Description

Loading a UTF8 KV file with Japanese (and perhaps other multi-byte?) text crashes Kivy under Windows. The example below uses the text field, but even a comment in Japanese will cause the crash. Changing the KV file encoding to Shift-JIS (approximately cp932) fixes the issue. 

Full details can be found in this user support group post: https://groups.google.com/forum/#!topic/kivy-users/dlR0f6tmANw

The essence is that, this fact combined with the fact that Linux-type systems support UTF8 KV files and NOT Shift-JIS KV files makes cross-platform development very difficult. 

Issue #4003 mentions a display glitch, but I believe a crash is more serious and may merit being dealt with. 

### Code and Logs

The following will reproduce the issue; the commented out alternatives are to show the options that work properly (without crashing).

utftest.py
```python
# -*- coding: utf-8 -*-

# test using kivy KV File in UTF8 with Japanese text

from kivy.app import App
from kivy.uix.boxlayout import BoxLayout
from kivy.lang import Builder

# # This will load properly (ignoring the fact that a font set needs to be loaded separately for proper display)
# Builder.load_string('''
# <UtfTestScreen>
#     Label:
#         # font_name: '../SourceHanSerifJP/SourceHanSerifJP-Medium.otf'
#         id: inst_txt
#         text: u'テストテキスト'
#
# ''')


class UtfTestScreen(BoxLayout):
    def __init__(self, **kwargs):
        super(UtfTestScreen, self).__init__(**kwargs)

        # # Test using Japanese text from py file to kv specified parts
        # self.ids.inst_txt.text = 'テストテキスト'


class UtfTestApp(App):
    def build(self):
        return UtfTestScreen()


if __name__ == "__main__":
    UtfTestApp().run()

```

utftest.kv **(saved in UTF8 encoding)**
```
<UtfTestScreen>
    Label:
        # font_name: '../SourceHanSerifJP/SourceHanSerifJP-Medium.otf'
        id: inst_txt
        text: u'テストテキスト'

```

caused an error as below:
```
 Traceback (most recent call last):
   File "utftest.py", line 34, in <module>
     UtfTestApp().run()
   File "C:\Python34\lib\site-packages\kivy\app.py", line 801, in run
     self.load_kv(filename=self.kv_file)
   File "C:\Python34\lib\site-packages\kivy\app.py", line 598, in load_kv
     root = Builder.load_file(rfilename)
   File "C:\Python34\lib\site-packages\kivy\lang.py", line 1831, in load_file
     data = fd.read()
 UnicodeDecodeError: 'cp932' codec can't decode byte 0x86 in position 141: illegal multibyte sequence

Process finished with exit code 1
```


Discussion:
The fix is easy enough, read in as bytes and decode to utf8 after reading. The question is whether we can always assume that the file is in utf8? Or maybe we need to have a way to specify the encoding like a shabang. Then by default it'll use the platform encoding, but it can be overwritten.
There is some similar discussion in #4003 , and I agree with VinGarcia's comment on 11 Apr 2016, to quote: 

> The best would probably be to mimic python behavior:
>    * Expect programmer to specify the encoding on the first line.
>    * Or else use python default encoding (on python3 it would always be 'utf8' I think).

If the default were to be made UTF8 on both Windows and Linux, or else the encoding be made specifiable as above, it'd really help me on this issue...
From the user support group above, @Geekademy shared an idea to use
```python
with open(filename, encoding='utf8') as f: 
    Builder.load_string(f.read()) 
```
in place of the usual `Builder.load_file()` , which seems to work properly for me. 
Although this is great as a temporary workaround, official Kivy Support of the above idea would be even greater... 
I have uploaded the minimal case, including points from this issue as comments: 

* https://github.com/yk0242/incubation/blob/master/incubation-python/kv-utf8/utftest.py
* https://github.com/yk0242/incubation/blob/master/incubation-python/kv-utf8/utftestkv.kv

Please feel free to use these for purposes of investigating this issue. 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

I found the same issue on answering an S.O. question yesterday.
SO, this _is_ actually a bug on Kivy, since  any app that is build even on systems that use utf-8 by default will fail to properly load non-ascii characters from the .KV on the "automatic" loading way, when they are run under windows.

To put it in other words: for any i18ned KV file, the app will break either when run on Windows, unless the .KV file uses the native windows encoding on the developer's computer. (in which case it won't work elsewhere)

IMHO, the fix would be forcing kivy to load .kv files using "utf-8" encoding and document that.

(I have no windows install, but the OP just accepted my answer, indicating this was indeed the problem - 
https://pt.stackoverflow.com/questions/281641/como-configurar-o-portugu%C3%AAs-no-kivy/281664#281664 )



Resolving patch:
diff --git a/kivy/lang/builder.py b/kivy/lang/builder.py
--- a/kivy/lang/builder.py
+++ b/kivy/lang/builder.py
@@ -285,7 +285,7 @@ def create_from(cls, builder):
 
         return obj
 
-    def load_file(self, filename, **kwargs):
+    def load_file(self, filename, encoding='utf8', **kwargs):
         '''Insert a file into the language builder and return the root widget
         (if defined) of the kv file.
 
@@ -293,14 +293,16 @@ def load_file(self, filename, **kwargs):
             `rulesonly`: bool, defaults to False
                 If True, the Builder will raise an exception if you have a root
                 widget inside the definition.
+
+            `encoding`: File charcter encoding. Defaults to utf-8,
         '''
         filename = resource_find(filename) or filename
         if __debug__:
-            trace('Lang: load file %s' % filename)
-        with open(filename, 'r') as fd:
-            kwargs['filename'] = filename
-            data = fd.read()
+            trace('Lang: load file %s, using %s encoding', filename, encoding)
 
+        kwargs['filename'] = filename
+        with open(filename, 'r', encoding=encoding) as fd:
+            data = fd.read()
             return self.load_string(data, **kwargs)
 
     def unload_file(self, filename):


Test patch:
diff --git a/kivy/tests/test_lang.py b/kivy/tests/test_lang.py
--- a/kivy/tests/test_lang.py
+++ b/kivy/tests/test_lang.py
@@ -4,6 +4,7 @@
 '''
 
 import unittest
+import os
 from weakref import proxy
 from functools import partial
 
@@ -294,6 +295,19 @@ def test_apply_rules(self):
         Builder.apply_rules(wid, 'TLangClassCustom')
         self.assertEqual(wid.obj, 42)
 
+    def test_load_utf8(self):
+        from tempfile import mkstemp
+        from kivy.lang import Builder
+        fd, name = mkstemp()
+        os.write(fd, '''
+
+Label:
+    text: 'é 😊'
+'''.encode('utf8'))
+        root = Builder.load_file(name)
+        assert root.text == 'é 😊'
+        os.close(fd)
+
 
 if __name__ == '__main__':
     unittest.main()