Repository: qutebrowser/qutebrowser

Issue:
UnicodeEncodeError with os.path.isdir in NetworkManager.createRequest
Report with `LC_ALL=C`:

```
10:56:42 DEBUG    init       app:process_pos_args:275 Startup URL file:///tmp/people.inf.elte.hu/mersaai/helpvizsgahoz/cpp%20vizsg%ED%B3%83%ED%B2%A1ra/index.html
[...]
10:56:42 ERROR    misc       utils:wrapper:616 Error in qutebrowser.browser.network.networkmanager.NetworkManager.createRequest
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/qutebrowser/utils/utils.py", line 614, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/qutebrowser/browser/network/networkmanager.py", line 365, in createRequest
    op, req, outgoing_data)
  File "/usr/lib/python3.5/site-packages/qutebrowser/browser/network/filescheme.py", line 116, in createRequest
    if os.path.isdir(path):
  File "/usr/lib/python3.5/genericpath.py", line 42, in isdir
    st = os.stat(s)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 55-60: ordinal not in range(128)
```



Discussion:
Hello. I would like to contribute for this issue. I think this is a very awesome tool and it would be great if I could make even a small contribution to this project. Could you please give me some pointers on how to begin? 

Hey @anantkaushik89! The first step would be to reproduce this issue on the newest git version by cloning the repo, and running `LC_ALL=C python3 -m qutebrowser --temp-basedir`, then navigating to a local file with non-ASCII characters in it.

If you still see the same crash, I'd recommend reading through the [contributing docs](https://github.com/The-Compiler/qutebrowser/blob/master/CONTRIBUTING.asciidoc) (though you can certainly ignore some irrelevant parts of it), and then getting the testsuite running with `tox` locally.

Then you can handle the `UnicodeEncodeError` correctly - the first thing I'd try would be simply returning `None` in that case, and see what Qt does when you visit the file. Alternatively, you could return an error page similar to the `error.html` returned from `dirbrowser_html` in the same file.

Let me know if something was unclear and you need more help!

Thanks so much @The-Compiler !! This does make life lot easier for me. I'll definitely get started and try to play around with the code base.

Hi @The-Compiler , can you help in one doubt? I installed qutebrower on my system. I tried opening files which have unicode characters in them like http://www.w3.org/2001/06/utf-8-test/postscript-utf-8.html. I also tried opening file in directories where the name contains something like %20 etc. I was able to open both these without issues.

Could you please tell a bit more on what the error was? Is it opening a local html file with unicode characters? Any sample file or url which you think can point me to the error would be very helpful.

Just to add, I am using :open command and then opening the file? Does the issue need some other steps in reproducing it?

You'll need to start qutebrowser with `LC_ALL=C` set (e.g. like above, with `LC_ALL=C python3 -m qutebrowser --temp-basedir`), to simulate systems with an ascii-only [locale](https://en.wikipedia.org/wiki/Locale_%28computer_software%29).

Hey @anantkaushik89, just found this issue again - were you able to reproduce it? Do you need help with something? Or did you move on to something else? :wink:
Hi @The-Compiler , you did guess it right :) I had tried back then to work this out and see the error but couldnt find it. I moved on to something else then 
@The-Compiler I can't reproduce this, running `LC_ALL=C python -m qutebrowser --temp-basedir foo.html` where `foo.html` is a local file with non-ascii characters.

Using `--backend webkit` displays garbage but does not raise an exception.

version info:
```
qutebrowser v1.0.2
Git commit: 378498bbd (2017-10-18 14:06:54 +0200)
Backend: QtWebEngine (Chromium 56.0.2924.122)

CPython: 3.6.2
Qt: 5.9.2
PyQt: 5.9

sip: 4.19.3
colorama: 0.3.9
pypeg2: 2.15
jinja2: 2.9.6
pygments: 2.2.0
yaml: 3.12
cssutils: 1.0.2 $Id$
attr: 17.2.0
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebKitWidgets: yes
pdf.js: no
sqlite: 3.20.1
QtNetwork SSL: OpenSSL 1.0.2l  25 May 2017

Style: QFusionStyle
Platform: Linux-4.13.7-1-ARCH-x86_64-with-arch, 64bit
Linux distribution: Arch Linux (arch)
Frozen: False
```
@7lb Note that this is about files with non-ascii characters in the *filename*, not the contents.
Oh, I missed that. I blame being tired. Will try to reproduce tomorrow if I have the time.

Resolving patch:
diff --git a/qutebrowser/browser/webkit/network/filescheme.py b/qutebrowser/browser/webkit/network/filescheme.py
--- a/qutebrowser/browser/webkit/network/filescheme.py
+++ b/qutebrowser/browser/webkit/network/filescheme.py
@@ -127,7 +127,10 @@ def createRequest(self, _op, request, _outgoing_data):
             A QNetworkReply for directories, None for files.
         """
         path = request.url().toLocalFile()
-        if os.path.isdir(path):
-            data = dirbrowser_html(path)
-            return networkreply.FixedDataNetworkReply(
-                request, data, 'text/html', self.parent())
+        try:
+            if os.path.isdir(path):
+                data = dirbrowser_html(path)
+                return networkreply.FixedDataNetworkReply(
+                    request, data, 'text/html', self.parent())
+        except UnicodeEncodeError:
+            return None


Test patch: