Repository: qutebrowser/qutebrowser

Issue:
Link with anchor on Qt 5.10 doesn't stop loading
The progressbar never goes away - probably a Qt 5.10 bug?

Looking at the logs, we get `loadFinished` and then another `loadStarted`?!

```
11:51:50 DEBUG    commands   command:run:484 command called: open ['https://docs.python.org/3/library/index.html#library-index']
[...]
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_url_changed(PyQt5.QtCore.QUrl('https://docs.python.org/3/library/index.html#library-index')) (tab 0)
[...]
11:51:50 DEBUG    webview    browsertab:_set_load_status:690 load status for <qutebrowser.browser.webengine.webenginetab.WebEngineTab tab_id=0 url='https://docs.python.org/3/library/index.html#library-index'>: LoadStatus.loading
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_status_changed('loading') (tab 0)
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_started() (tab 0)
[...]
11:51:50 DEBUG    webview    webview:acceptNavigationRequest:299 navigation request: url https://docs.python.org/3/library/index.html#library-index, type NavigationTypeTyped, is_main_frame True
[...]
11:51:50 DEBUG    webview    browsertab:_set_load_status:690 load status for <qutebrowser.browser.webengine.webenginetab.WebEngineTab tab_id=0 url='https://docs.python.org/3/library/index.html#library-index'>: LoadStatus.success_https
[...]
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_status_changed('success_https') (tab 0)
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_finished(True) (tab 0)
11:51:50 DEBUG    webview    browsertab:_set_load_status:690 load status for <qutebrowser.browser.webengine.webenginetab.WebEngineTab tab_id=0 url='https://docs.python.org/3/library/index.html#library-index'>: LoadStatus.loading
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_status_changed('loading') (tab 0)
11:51:50 DEBUG    signals    signalfilter:_filter_signals:86 emitting: cur_load_started() (tab 0)
```



Discussion:
I opened [QTBUG-65223](https://bugreports.qt.io/browse/QTBUG-65223).
I can reproduce this without an anchor in the URL:

1. Open a new qutebrowser window
1. Navigate to github.com with `ogithub.com/qutebrowser/qutebrowser<enter>`
1. Go to the PR list, either by clicking or via hints

Installed qutebrowser via the `qutebrowser-git` AUR package this morning.

`:version` output
```
qutebrowser v1.0.4
Git commit: 8173a48b8 (2017-12-14 22:56:44 +0100)
Backend: QtWebEngine (Chromium 61.0.3163.140)

CPython: 3.6.3
Qt: 5.10.0
PyQt: 5.9.2

sip: 4.19.6
colorama: no
pypeg2: 2.15
jinja2: 2.10
pygments: 2.2.0
yaml: 3.12
cssutils: no
attr: 17.3.0
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebKitWidgets: no
pdf.js: no
sqlite: 3.21.0
QtNetwork SSL: OpenSSL 1.1.0g  2 Nov 2017

Style: QFusionStyle
Platform: Linux-4.14.5-1-ARCH-x86_64-with-arch, 64bit
Linux distribution: Arch Linux (arch)
Frozen: False
Imported from /usr/lib/python3.6/site-packages/qutebrowser
Qt library executable path: /usr/lib/qt/libexec, data path: /usr/share/qt

Paths:
cache: /home/olmo/.cache/qutebrowser
config: /home/olmo/.config/qutebrowser
data: /home/olmo/.local/share/qutebrowser
runtime: /run/user/1000/qutebrowser
system data: /usr/share/qutebrowser

Uptime: 9:47:34
```
It's potentially possible to mitigate this by relying on `loadProgress(100)` in place of `loadFinished(True)`. As a quick-and-dirty proof of concept: 
```
diff --git a/qutebrowser/browser/browsertab.py b/qutebrowser/browser/browsertab.py
index eb0e55c4b..f30da0240 100644
--- a/qutebrowser/browser/browsertab.py
+++ b/qutebrowser/browser/browsertab.py
@@ -749,23 +749,15 @@ class AbstractTab(QWidget):
 
     @pyqtSlot(bool)
     def _on_load_finished(self, ok):
-        sess_manager = objreg.get('session-manager')
-        sess_manager.save_autosave()
-
-        if ok and not self._has_ssl_errors:
-            if self.url().scheme() == 'https':
-                self._set_load_status(usertypes.LoadStatus.success_https)
-            else:
-                self._set_load_status(usertypes.LoadStatus.success)
-        elif ok:
-            self._set_load_status(usertypes.LoadStatus.warn)
-        else:
+        if not ok:
+            sess_manager = objreg.get('session-manager')
+            sess_manager.save_autosave()
             self._set_load_status(usertypes.LoadStatus.error)
-        self.load_finished.emit(ok)
-        if not self.title():
-            self.title_changed.emit(self.url().toDisplayString())
+            self.load_finished.emit(ok)
+            if not self.title():
+                self.title_changed.emit(self.url().toDisplayString())
 
-        self.zoom.set_current()
+            self.zoom.set_current()
 
     @pyqtSlot()
     def _on_history_trigger(self):
@@ -777,6 +769,23 @@ class AbstractTab(QWidget):
         self._progress = perc
         self.load_progress.emit(perc)
 
+        if perc == 100 and self.load_status() == usertypes.LoadStatus.loading:
+            sess_manager = objreg.get('session-manager')
+            sess_manager.save_autosave()
+
+            if not self._has_ssl_errors:
+                if self.url().scheme() == 'https':
+                    self._set_load_status(usertypes.LoadStatus.success_https)
+                else:
+                    self._set_load_status(usertypes.LoadStatus.success)
+            else:
+                self._set_load_status(usertypes.LoadStatus.warn)
+            self.load_finished.emit(True)
+            if not self.title():
+                self.title_changed.emit(self.url().toDisplayString())
+
+            self.zoom.set_current()
+
     @pyqtSlot()
     def _on_ssl_errors(self):
         self._has_ssl_errors = True
```

That leaves quite a few sites now subjected to #3110, and is dependent on `loadFinished(False)` being emitted for all unrecoverable errors (which is not a particularly sound assumption when the flakiness of `loadFinished` is the only reason this problem exists) but it does stop the persistent progress bar.

Resolving patch:
diff --git a/qutebrowser/browser/webengine/webenginetab.py b/qutebrowser/browser/webengine/webenginetab.py
--- a/qutebrowser/browser/webengine/webenginetab.py
+++ b/qutebrowser/browser/webengine/webenginetab.py
@@ -24,7 +24,8 @@
 import html as html_utils
 
 import sip
-from PyQt5.QtCore import pyqtSlot, Qt, QEvent, QPoint, QPointF, QUrl, QTimer
+from PyQt5.QtCore import (pyqtSignal, pyqtSlot, Qt, QEvent, QPoint, QPointF,
+                          QUrl, QTimer)
 from PyQt5.QtGui import QKeyEvent
 from PyQt5.QtNetwork import QAuthenticator
 from PyQt5.QtWidgets import QApplication
@@ -539,7 +540,15 @@ def find_at_pos(self, pos, callback):
 
 class WebEngineTab(browsertab.AbstractTab):
 
-    """A QtWebEngine tab in the browser."""
+    """A QtWebEngine tab in the browser.
+
+    Signals:
+        _load_finished_fake:
+            Used in place of unreliable loadFinished
+    """
+
+    # WORKAROUND for https://bugreports.qt.io/browse/QTBUG-65223
+    _load_finished_fake = pyqtSignal(bool)
 
     def __init__(self, *, win_id, mode_manager, private, parent=None):
         super().__init__(win_id=win_id, mode_manager=mode_manager,
@@ -793,6 +802,24 @@ def _on_render_process_terminated(self, status, exitcode):
         }
         self.renderer_process_terminated.emit(status_map[status], exitcode)
 
+    @pyqtSlot(int)
+    def _on_load_progress_workaround(self, perc):
+        """Use loadProgress(100) to emit loadFinished(True).
+
+        See https://bugreports.qt.io/browse/QTBUG-65223
+        """
+        if perc == 100 and self.load_status() != usertypes.LoadStatus.error:
+            self._load_finished_fake.emit(True)
+
+    @pyqtSlot(bool)
+    def _on_load_finished_workaround(self, ok):
+        """Use only loadFinished(False).
+
+        See https://bugreports.qt.io/browse/QTBUG-65223
+        """
+        if not ok:
+            self._load_finished_fake.emit(False)
+
     def _connect_signals(self):
         view = self._widget
         page = view.page()
@@ -801,9 +828,6 @@ def _connect_signals(self):
         page.linkHovered.connect(self.link_hovered)
         page.loadProgress.connect(self._on_load_progress)
         page.loadStarted.connect(self._on_load_started)
-        page.loadFinished.connect(self._on_history_trigger)
-        page.loadFinished.connect(self._restore_zoom)
-        page.loadFinished.connect(self._on_load_finished)
         page.certificate_error.connect(self._on_ssl_errors)
         page.authenticationRequired.connect(self._on_authentication_required)
         page.proxyAuthenticationRequired.connect(
@@ -816,6 +840,19 @@ def _connect_signals(self):
         view.renderProcessTerminated.connect(
             self._on_render_process_terminated)
         view.iconChanged.connect(self.icon_changed)
+        # WORKAROUND for https://bugreports.qt.io/browse/QTBUG-65223
+        if qtutils.version_check('5.10', compiled=False):
+            page.loadProgress.connect(self._on_load_progress_workaround)
+            self._load_finished_fake.connect(self._on_history_trigger)
+            self._load_finished_fake.connect(self._restore_zoom)
+            self._load_finished_fake.connect(self._on_load_finished)
+            page.loadFinished.connect(self._on_load_finished_workaround)
+        else:
+            # for older Qt versions which break with the above
+            page.loadProgress.connect(self._on_load_progress)
+            page.loadFinished.connect(self._on_history_trigger)
+            page.loadFinished.connect(self._restore_zoom)
+            page.loadFinished.connect(self._on_load_finished)
 
     def event_target(self):
         return self._widget.focusProxy()


Test patch:
diff --git a/tests/end2end/features/marks.feature b/tests/end2end/features/marks.feature
--- a/tests/end2end/features/marks.feature
+++ b/tests/end2end/features/marks.feature
@@ -86,7 +86,7 @@ Feature: Setting positional marks
         And I wait until the scroll position changed to 10/10
         Then the page should be scrolled to 10 10
 
-    @qtwebengine_todo: Does not emit loaded signal for fragments?
+    @qtwebengine_skip: Does not emit loaded signal for fragments?
     Scenario: Jumping back after following a link
         When I hint with args "links normal" and follow s
         And I wait until data/marks.html#bottom is loaded