Repository: qutebrowser/qutebrowser

Issue:
Issues with gU on qute://help/
Doing `gU` on `qute://help/quickstart.html` tries to open `qute://help/quickstart.html/..`.

In https://crashes.qutebrowser.org/view/a07c404e this also lead to an `AssertionError` on 984dd1ba8:

```
commands   command:run:489 command called: navigate ['up', '-t']
commands   command:run:504 Calling qutebrowser.browser.commands.CommandDispatcher.navigate(<qutebrowser.browser.commands.CommandDispatcher>, 'up', True, False, False, 1)
webview    tabbedbrowser:tabopen:445 Creating new tab with URL PyQt5.QtCore.QUrl('qute://help/img/cheatsheet-big.png/..'), background False, related True, idx None
[...]
webview    webview:acceptNavigationRequest:296 navigation request: url qute://help/img/cheatsheet-big.png/.., type NavigationTypeTyped, is_main_frame True
misc       webenginequtescheme:requestStarted:50 Got request for qute://help/img/cheatsheet-big.png/..
misc       qutescheme:data_for_url:143 url: qute://help/img/cheatsheet-big.png/.., path: /img/cheatsheet-big.png/.., host help
misc       crashsignal:exception_hook:216 Uncaught exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/webengine/webenginequtescheme.py", line 52, in requestStarted
    mimetype, data = qutescheme.data_for_url(url)
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/qutescheme.py", line 158, in data_for_url
    mimetype, data = handler(url)
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/qutescheme.py", line 118, in wrapper
    return self._function(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/qutebrowser/browser/qutescheme.py", line 329, in qute_help
    assert mimetype is not None, url
AssertionError: PyQt5.QtCore.QUrl('qute://help/img/cheatsheet-big.png/..')
```



Discussion:
What should be the behaviour when doing gu/gU on a url like qute://help/img/cheatsheet-big.png? Should it  
go to qute://help/ or go to qute://help/img and show an error message that says "Couldn't open file" or something like that.
I think it should (try to) go to `qute://help/img`.
I came up with this solution and want your feedback on it :)

```diff
diff --git a/qutebrowser/browser/qutescheme.py b/qutebrowser/browser/qutescheme.py
index e5fd470a1..b1121e8f4 100644
--- a/qutebrowser/browser/qutescheme.py
+++ b/qutebrowser/browser/qutescheme.py
@@ -135,6 +135,11 @@ def data_for_url(url):
     Return:
         A (mimetype, data) tuple.
     """
+    # normalize path and strip trailing slashes
+    norm_url = url.adjusted(QUrl.NormalizePathSegments | QUrl.StripTrailingSlash)
+    if norm_url != url:
+        raise Redirect(norm_url)
+
     path = url.path()
     host = url.host()
     query = urlutils.query_string(url)
@@ -332,8 +337,12 @@ def qute_help(url):
     path = 'html/doc/{}'.format(urlpath)
     if not urlpath.endswith('.html'):
         mimetype, _encoding = mimetypes.guess_type(urlpath)
+        try:
+            bdata = utils.read_file(path, binary=True)
+        except OSError as e:
+            raise QuteSchemeOSError(e)
         assert mimetype is not None, url
-        return mimetype, utils.read_file(path, binary=True)
+        return mimetype, bdata
 
     try:
         data = utils.read_file(path)

```
The lower part looks good (assuming that `utils.read_file` raises `OSError` on directories).

As for the upper part: I think this might affect other sites as well, and should be fixed in `qutebrowser/browser/navigate.py` by normalizing the path there instead.
> The lower part looks good (assuming that utils.read_file raises OSError on directories).

Yes, utils.read_file returns IsADirectoryError (a subclass of OSError) on directories.

> As for the upper part: I think this might affect other sites as well, 

I thought the problem was only on qute://* sites as we have to manually handle the path and make sense of all the '..' and trailing slashes, whereas in http sites, it is taken care by the web server itself. Am I mistaken?

> and should be fixed in qutebrowser/browser/navigate.py by normalizing the path there instead.

you mean inside path_up() function?



> whereas in http sites, it is taken care by the web server itself.

That's *probably* true, but we can't know for sure - thus I think it'd be nicer if qutebrowser did the right thing.

> you mean inside path_up() function?

Right!
``` diff
diff --git a/qutebrowser/browser/navigate.py b/qutebrowser/browser/navigate.py
index 2278a2fa7..8b406368b 100644
--- a/qutebrowser/browser/navigate.py
+++ b/qutebrowser/browser/navigate.py
@@ -64,6 +64,7 @@ def path_up(url, count):
         raise Error("Can't go up!")
     for _i in range(0, min(count, path.count('/'))):
         path = posixpath.join(path, posixpath.pardir)
+    path = posixpath.normpath(path)
     url.setPath(path)
     return url
```
This will fix the gu/gU command, however, if a user manually typed a url like ```qute://help/img/../contributing.html``` or ```qute://help//commands.html/```, qb won't open the correct page. I wanted to sanitize each and every url before sending them out  :smile: 
Fair, let's do both then 😆 

Why `QUrl.StripTrailingSlash` though? Wouldn't that just do another redirect to add the trailing slash then?
```QUrl.StripTrailingSlash``` removes only redundant slashes e.g. it will not remove the trailing slash in ```qute://help/``` because the path resolves to just  '/', but it will remove the trailing slash from ```qute://help/img/``` as the path resolves to '/img/' where the last slash is redundant. 

The [documentation](https://doc.qt.io/qt-5/qurl.html#UrlFormattingOption-enum) on this could be more clear though.

Shall I submit a PR then? 
Should I add tests for this in ```tests/end2end/features/navigate.feature``` or in ```tests/end2end/features/qutescheme.feature``` ? Also, how do I write a scenario for a url like ```qute://help/img``` where qb actually fails to load the page.
Sorry for the late answer! Yeah, please open a PR.

I think it'd be good to add tests in both - in `navigate.feature` to make sure it normalizes the URL, and in `qutescheme.feature` to make sure it handles paths with `../` parts in it.

As for testing for the load failure: I usually do `Then "foobarbaz" should be logged`, find the respective log line in the output, and then adjust the text accordingly. If you can't make it work, feel free to submit it with a work-in-progress test and I'll take a look.

Resolving patch:
diff --git a/qutebrowser/browser/navigate.py b/qutebrowser/browser/navigate.py
--- a/qutebrowser/browser/navigate.py
+++ b/qutebrowser/browser/navigate.py
@@ -64,6 +64,7 @@ def path_up(url, count):
         raise Error("Can't go up!")
     for _i in range(0, min(count, path.count('/'))):
         path = posixpath.join(path, posixpath.pardir)
+    path = posixpath.normpath(path)
     url.setPath(path)
     return url
 
diff --git a/qutebrowser/browser/qutescheme.py b/qutebrowser/browser/qutescheme.py
--- a/qutebrowser/browser/qutescheme.py
+++ b/qutebrowser/browser/qutescheme.py
@@ -135,6 +135,11 @@ def data_for_url(url):
     Return:
         A (mimetype, data) tuple.
     """
+    norm_url = url.adjusted(QUrl.NormalizePathSegments |
+            QUrl.StripTrailingSlash)
+    if norm_url != url:
+        raise Redirect(norm_url)
+
     path = url.path()
     host = url.host()
     query = urlutils.query_string(url)
@@ -331,9 +336,13 @@ def qute_help(url):
 
     path = 'html/doc/{}'.format(urlpath)
     if not urlpath.endswith('.html'):
+        try:
+            bdata = utils.read_file(path, binary=True)
+        except OSError as e:
+            raise QuteSchemeOSError(e)
         mimetype, _encoding = mimetypes.guess_type(urlpath)
         assert mimetype is not None, url
-        return mimetype, utils.read_file(path, binary=True)
+        return mimetype, bdata
 
     try:
         data = utils.read_file(path)


Test patch:
diff --git a/tests/end2end/features/navigate.feature b/tests/end2end/features/navigate.feature
--- a/tests/end2end/features/navigate.feature
+++ b/tests/end2end/features/navigate.feature
@@ -18,6 +18,11 @@ Feature: Using :navigate
         And I run :navigate up with count 2
         Then data/navigate should be loaded
 
+    Scenario: Navigating up in qute://help/
+        When I open qute://help/commands.html
+        And I run :navigate up
+        Then qute://help/ should be loaded
+
     # prev/next
 
     Scenario: Navigating to previous page
diff --git a/tests/end2end/features/qutescheme.feature b/tests/end2end/features/qutescheme.feature
--- a/tests/end2end/features/qutescheme.feature
+++ b/tests/end2end/features/qutescheme.feature
@@ -63,6 +63,22 @@ Feature: Special qute:// pages
         And I hint with args "links normal" and follow a
         Then qute://help/quickstart.html should be loaded
 
+    Scenario: Opening a link with qute://help/index.html/..
+        When the documentation is up to date
+        And I open qute://help/index.html/.. without waiting
+        Then qute://help/ should be loaded
+
+    Scenario: Opening a link with qute://help/index.html/../
+        When the documentation is up to date
+        And I open qute://help/index.html/../ without waiting
+        Then qute://help/ should be loaded
+
+    Scenario: Opening a link with qute://help/img/
+        When the documentation is up to date
+        And I open qute://help/img/ without waiting
+        Then "OSError while handling qute://* URL" should be logged
+        And "* url='qute://help/img'* LoadStatus.error" should be logged
+
     # :history
 
     Scenario: :history without arguments