Repository: qutebrowser/qutebrowser
Pull request: Re-install all modules every time a new profile is created.
Alright, this is not a proper fix, but just so we know which is the cause and where we need to start. All that `something.install(filter)` logic that takes place in `qutebrowser/browser/webengine/webenginetab.py` needs to be repeated every time we create a new profile. Maybe all of that should be moved to the respective `init_X_profile` functions?

Also, while testing if this is exclusive to QtWebEngine, I'm constantly hitting a segmentation fault when trying these steps with QtWebkit and my local newspaper website: https://elpais.com.uy. Not really related but I figured I should mention it.

Fixes #5810


Problem statement:
Adblocking stops working when last private window is closed
**Version info**:
<!-- Please copy the first block from :version, not just the qutebrowser version -->
qutebrowser v1.14.0
Git commit: 
Backend: QtWebEngine (Chromium 80.0.3987.163)
Qt: 5.15.1

**Does the bug happen if you start with `--temp-basedir`?**:
Yes.

**Description**
Doing _some_ sequence of actions adblocking stops working.

**How to reproduce**
<!-- Link to the affected site, or steps to reproduce the issue (if possible/applicable). -->
Took me a while to figure out reproducer sequence, might as well affect different one:
1. Start qutebrowser with adblock list updated
2. Open private window :open -p
3. Go to page with ads (that's basically any web page now)
4. Confirm ads are not shown
5. Close private window
6. Open private window again :open -p
7. Go to the same web page
8. Ads are not blocked

Hints from issue discussion:
This happened somewhere between 786198fde5bae03b003768f84bf577f718fd784d (good) and 35d116dd43ce131f9ac685826f073c698c030ca2 (bad), so was introduced by #4147 (or my changes after that).

cc @lufte
Wow, great finding @jpalus. I can take a look this weekend.
Looks like it affects downloads as well -- selecting downloadable link in second private window does nothing.
It affects content.javascript.enabled as well (enables JavaScript even if it's previously set to disabled), and the linked pull request did not fix it.

The reproduce steps are roughly similar to the one described in the issue.
@user202729 Reproduced with

```
qutebrowser --temp-basedir -s content.javascript.enabled false about:blank ':open -p https://www.whatismybrowser.com/detect/is-javascript-enabled'`
```

then closing the private window, and doing

```
:open -p https://www.whatismybrowser.com/detect/is-javascript-enabled
```
again.

Also seems to be caused by a commit between 786198f (good) and 35d116d (bad) - and indeed neither #5813 nor https://github.com/lufte/qutebrowser/commit/140c728cc475a929a22e2783a535304a94474e09 fixes it...

I haven't really looked into it yet, and I have no idea why this could be happening, given that JS is a per-page rather than per-profile setting. @lufte do you have any idea what could be going on there?

edit: Ah, I think the problem is that we never call `global_settings.init_settings()` after swapping out the profile, thus never applying the settings as long as they haven't changed...
Uh-oh.

```python
    global_settings.init_settings()

    _init_site_specific_quirks()
    _init_devtools_settings()
```
This is what's called after initializing the profiles in `webenginesettings.init`. After reviewing this issue I assumed we needed to call all 3 of these after creating a new private profile, however:
1. The custom user agents set in `_init_site_specific_quirks` are preserved. Is it because the user agent is read by qutebrowser directly from the config (by the interceptor module) instead of QWebEngine reading it from the profile?
2. The custom javascript.enabled values set in `_init_devtools_settings` are also preserved. Is it because settings set with a pattern are not affected by this issue? I tested your steps but setting `content.javascript.enabled` only for `https://www.whatismybrowser.com/*`, and it was preserved. I couldn't figure out this behavior.

Assuming that we only need `global_settings.init_settings()`, I think the following patch is enough:
```
diff --git a/qutebrowser/config/websettings.py b/qutebrowser/config/websettings.py
index cc307dd75..0f4616c4b 100644
--- a/qutebrowser/config/websettings.py
+++ b/qutebrowser/config/websettings.py
@@ -287,6 +287,7 @@ def clear_private_data() -> None:
     if objects.backend == usertypes.Backend.QtWebEngine:
         from qutebrowser.browser.webengine import webenginesettings
         webenginesettings.init_private_profile()
+        webenginesettings.global_settings.init_settings()
     elif objects.backend == usertypes.Backend.QtWebKit:
         from qutebrowser.browser.webkit import cookies
         assert cookies.ram_cookie_jar is not None
```

I'll push all my pending changes to the current PR so they are not scattered all over the place. If we decide we still want the first approach (the one with `if not is_first_time:`) I'll make the appropriate changes.

Commit message: Remove unused import

Patch to review:
diff --git a/qutebrowser/app.py b/qutebrowser/app.py
index f722c580f78..8f96509241a 100644
--- a/qutebrowser/app.py
+++ b/qutebrowser/app.py
@@ -461,10 +461,6 @@ def _init_modules(*, args):
     log.init.debug("Initializing sessions...")
     sessions.init(q_app)
 
-    log.init.debug("Initializing websettings...")
-    websettings.init(args)
-    quitter.instance.shutting_down.connect(websettings.shutdown)
-
     if not args.no_err_windows:
         crashsignal.crash_handler.display_faulthandler()
 
@@ -479,6 +475,10 @@ def _init_modules(*, args):
     log.init.debug("Initializing cookies...")
     cookies.init(q_app)
 
+    log.init.debug("Initializing websettings...")
+    websettings.init(args)
+    quitter.instance.shutting_down.connect(websettings.shutdown)
+
     log.init.debug("Initializing cache...")
     cache.init(q_app)
 
@@ -492,7 +492,6 @@ def _init_modules(*, args):
     macros.init()
     windowundo.init()
     # Init backend-specific stuff
-    browsertab.init()
 
 
 class Application(QApplication):
diff --git a/qutebrowser/browser/browsertab.py b/qutebrowser/browser/browsertab.py
index dd21cd0e026..bf25455a624 100644
--- a/qutebrowser/browser/browsertab.py
+++ b/qutebrowser/browser/browsertab.py
@@ -83,15 +83,6 @@ def create(win_id: int,
                      parent=parent)
 
 
-def init() -> None:
-    """Initialize backend-specific modules."""
-    if objects.backend == usertypes.Backend.QtWebEngine:
-        from qutebrowser.browser.webengine import webenginetab
-        webenginetab.init()
-        return
-    assert objects.backend == usertypes.Backend.QtWebKit, objects.backend
-
-
 class WebTabError(Exception):
 
     """Base class for various errors."""
diff --git a/qutebrowser/browser/webengine/webenginesettings.py b/qutebrowser/browser/webengine/webenginesettings.py
index 1526574a718..e28250706a6 100644
--- a/qutebrowser/browser/webengine/webenginesettings.py
+++ b/qutebrowser/browser/webengine/webenginesettings.py
@@ -29,12 +29,17 @@
 from typing import cast, Any, List, Optional, Tuple, Union
 
 from PyQt5.QtGui import QFont
+from PyQt5.QtWidgets import QApplication
 from PyQt5.QtWebEngineWidgets import QWebEngineSettings, QWebEngineProfile
 
-from qutebrowser.browser.webengine import spell, webenginequtescheme
+from qutebrowser.browser import history
+from qutebrowser.browser.webengine import (spell, webenginequtescheme, cookies,
+                                           webenginedownloads)
 from qutebrowser.config import config, websettings
 from qutebrowser.config.websettings import AttributeInfo as Attr
-from qutebrowser.utils import standarddir, qtutils, message, log, urlmatch, usertypes
+from qutebrowser.utils import (standarddir, qtutils, message, log,
+                               urlmatch, usertypes, objreg)
+
 
 # The default QWebEngineProfile
 default_profile = cast(QWebEngineProfile, None)
@@ -45,6 +50,10 @@
 
 parsed_user_agent = None
 
+_qute_scheme_handler = cast(webenginequtescheme.QuteSchemeHandler, None)
+_req_interceptor = None
+_download_manager = cast(webenginedownloads.DownloadManager, None)
+
 
 class _SettingsWrapper:
 
@@ -344,6 +353,16 @@ def _init_default_profile():
     default_profile.setter.init_profile()
     default_profile.setter.set_persistent_cookie_policy()
 
+    _qute_scheme_handler.install(default_profile)
+    _req_interceptor.install(default_profile)  # type: ignore
+    _download_manager.install(default_profile)
+    cookies.install_filter(default_profile)
+
+    # Clear visited links on web history clear
+    history.web_history.history_cleared.connect(default_profile.clearAllVisitedLinks)
+    history.web_history.url_cleared.connect(
+        lambda url, profile=default_profile: profile.clearVisitedLinks([url]))
+
 
 def init_private_profile():
     """Init the private QWebEngineProfile."""
@@ -356,6 +375,17 @@ def init_private_profile():
         assert private_profile.isOffTheRecord()
         private_profile.setter.init_profile()
 
+        _qute_scheme_handler.install(private_profile)
+        _req_interceptor.install(private_profile)  # type: ignore
+        _download_manager.install(private_profile)
+        cookies.install_filter(private_profile)
+
+        # Clear visited links on web history clear
+        history.web_history.history_cleared.connect(
+            default_profile.clearAllVisitedLinks)
+        history.web_history.url_cleared.connect(
+            lambda url, profile=private_profile: profile.clearVisitedLinks([url]))
+
 
 def _init_site_specific_quirks():
     """Add custom user-agent settings for problematic sites.
@@ -430,6 +460,26 @@ def init():
     webenginequtescheme.init()
     spell.init()
 
+    # For some reason we need to keep a reference, otherwise the scheme handler
+    # won't work...
+    # https://www.riverbankcomputing.com/pipermail/pyqt/2016-September/038075.html
+    global _qute_scheme_handler
+    app = QApplication.instance()
+    log.init.debug("Initializing qute://* handler...")
+    _qute_scheme_handler = webenginequtescheme.QuteSchemeHandler(parent=app)
+
+    global _req_interceptor
+    log.init.debug("Initializing request interceptor...")
+    from qutebrowser.browser.webengine import interceptor
+    _req_interceptor = interceptor.RequestInterceptor(parent=app)
+
+    global _download_manager
+    log.init.debug("Initializing QtWebEngine downloads...")
+    _download_manager = webenginedownloads.DownloadManager(parent=app)
+    objreg.register('webengine-download-manager', _download_manager)
+    from qutebrowser.misc import quitter
+    quitter.instance.shutting_down.connect(_download_manager.shutdown)
+
     _init_default_profile()
     init_private_profile()
     config.instance.changed.connect(_update_settings)
diff --git a/qutebrowser/browser/webengine/webenginetab.py b/qutebrowser/browser/webengine/webenginetab.py
index e228091f434..21c6250e4a8 100644
--- a/qutebrowser/browser/webengine/webenginetab.py
+++ b/qutebrowser/browser/webengine/webenginetab.py
@@ -27,68 +27,19 @@
 
 from PyQt5.QtCore import pyqtSignal, pyqtSlot, Qt, QPoint, QPointF, QUrl, QObject
 from PyQt5.QtNetwork import QAuthenticator
-from PyQt5.QtWidgets import QApplication, QWidget
+from PyQt5.QtWidgets import QWidget
 from PyQt5.QtWebEngineWidgets import QWebEnginePage, QWebEngineScript, QWebEngineHistory
 
 from qutebrowser.config import config
-from qutebrowser.browser import (browsertab, eventfilter, shared, webelem,
-                                 history, greasemonkey)
+from qutebrowser.browser import (browsertab, eventfilter, shared, webelem, greasemonkey)
 from qutebrowser.browser.webengine import (webview, webengineelem, tabhistory,
-                                           interceptor, webenginequtescheme,
-                                           cookies, webenginedownloads,
                                            webenginesettings, certificateerror)
-from qutebrowser.misc import miscwidgets, objects, quitter
+from qutebrowser.misc import miscwidgets, objects
 from qutebrowser.utils import (usertypes, qtutils, log, javascript, utils,
-                               message, objreg, jinja, debug)
+                               message, jinja, debug)
 from qutebrowser.qt import sip
 
 
-_qute_scheme_handler = None
-
-
-def init():
-    """Initialize QtWebEngine-specific modules."""
-    # For some reason we need to keep a reference, otherwise the scheme handler
-    # won't work...
-    # https://www.riverbankcomputing.com/pipermail/pyqt/2016-September/038075.html
-    global _qute_scheme_handler
-
-    app = QApplication.instance()
-    log.init.debug("Initializing qute://* handler...")
-    _qute_scheme_handler = webenginequtescheme.QuteSchemeHandler(parent=app)
-    _qute_scheme_handler.install(webenginesettings.default_profile)
-    if webenginesettings.private_profile:
-        _qute_scheme_handler.install(webenginesettings.private_profile)
-
-    log.init.debug("Initializing request interceptor...")
-    req_interceptor = interceptor.RequestInterceptor(parent=app)
-    req_interceptor.install(webenginesettings.default_profile)
-    if webenginesettings.private_profile:
-        req_interceptor.install(webenginesettings.private_profile)
-
-    log.init.debug("Initializing QtWebEngine downloads...")
-    download_manager = webenginedownloads.DownloadManager(parent=app)
-    download_manager.install(webenginesettings.default_profile)
-    if webenginesettings.private_profile:
-        download_manager.install(webenginesettings.private_profile)
-    objreg.register('webengine-download-manager', download_manager)
-    quitter.instance.shutting_down.connect(download_manager.shutdown)
-
-    log.init.debug("Initializing cookie filter...")
-    cookies.install_filter(webenginesettings.default_profile)
-    if webenginesettings.private_profile:
-        cookies.install_filter(webenginesettings.private_profile)
-
-    # Clear visited links on web history clear
-    for p in [webenginesettings.default_profile,
-              webenginesettings.private_profile]:
-        if not p:
-            continue
-        history.web_history.history_cleared.connect(p.clearAllVisitedLinks)
-        history.web_history.url_cleared.connect(
-            lambda url, profile=p: profile.clearVisitedLinks([url]))
-
-
 # Mapping worlds from usertypes.JsWorld to QWebEngineScript world IDs.
 _JS_WORLD_MAP = {
     usertypes.JsWorld.main: QWebEngineScript.MainWorld,
diff --git a/qutebrowser/config/websettings.py b/qutebrowser/config/websettings.py
index e1cb393dcd4..2b76ab9c53d 100644
--- a/qutebrowser/config/websettings.py
+++ b/qutebrowser/config/websettings.py
@@ -247,6 +247,7 @@ def clear_private_data() -> None:
     if objects.backend == usertypes.Backend.QtWebEngine:
         from qutebrowser.browser.webengine import webenginesettings
         webenginesettings.init_private_profile()
+        webenginesettings.global_settings.init_settings()
     elif objects.backend == usertypes.Backend.QtWebKit:
         from qutebrowser.browser.webkit import cookies
         assert cookies.ram_cookie_jar is not None


Code review comments:
diff_hunk: @@ -479,6 +475,10 @@ def _init_modules(*, args):
     log.init.debug("Initializing cookies...")
     cookies.init(q_app)
 
+    log.init.debug("Initializing websettings...")
+    websettings.init(args)
+    quitter.instance.shutting_down.connect(websettings.shutdown)
path: qutebrowser/app.py
text: @user1:
Can you elaborate why you moved this down? From a quick look, things still seem properly when moving it up again, and the init sequence is somewhat brittle.

@author:
I remember specifically moving it because the `webenginesettings` module now makes use of other modules that needed to be initialized first, but now it looks like it was a confusion. I moved it below the initialization of the `cookies` module, but that's not the same `cookies` module used by `webenginesettings`. And you're right, switching it back doesn't break anything. 

Hmm, maybe there was an issue and I thought I was fixing it with that; but actually I fixed it with something else and forgot to revert the first change...

I'll switch it back.

diff_hunk: @@ -27,68 +27,19 @@
 
 from PyQt5.QtCore import pyqtSignal, pyqtSlot, Qt, QPoint, QPointF, QUrl, QObject
 from PyQt5.QtNetwork import QAuthenticator
-from PyQt5.QtWidgets import QApplication, QWidget
+from PyQt5.QtWidgets import QWidget
 from PyQt5.QtWebEngineWidgets import QWebEnginePage, QWebEngineScript, QWebEngineHistory
 
 from qutebrowser.config import config
-from qutebrowser.browser import (browsertab, eventfilter, shared, webelem,
-                                 history, greasemonkey)
+from qutebrowser.browser import (browsertab, eventfilter, shared, webelem, greasemonkey)
path: qutebrowser/browser/webengine/webenginetab.py
text: @user1:
nitpick:

```suggestion
from qutebrowser.browser import browsertab, eventfilter, shared, webelem, greasemonkey
```

diff_hunk: @@ -45,6 +50,10 @@
 
 parsed_user_agent = None
 
+_qute_scheme_handler = cast(webenginequtescheme.QuteSchemeHandler, None)
+_req_interceptor = None
path: qutebrowser/browser/webengine/webenginesettings.py
text: @user1:
```suggestion
_req_interceptor = cast(interceptor.RequestInterceptor, None)
```

then we can get rid of the `# type: ignore` comments because we essentially tell mypy "even though we set this to `None`, it's initialized to a value very early, so you can expect this to never actually be `None`"

@author:
The problem is that `interceptor` is not imported at that point, it is imported later on in `webenginesettings.init`. mypy complains about that, and I coulnd't find a way of using something as a type before it is actually imported.

@user1:
You can pass it as a string instead, and add the import under a `if TYPE_CHECKING:` (via `from typing import TYPE_CHECKING`)

@author:
Ah, perfect, thanks. Like this? b1bf969478e4b64d1f0871fd402ac5822d06286c