Repository: qutebrowser/qutebrowser

Issue:
Host-based subdomain blocking breaks search functionality of AWS console
**Version info**:
```
qutebrowser v2.3.0
Git commit: 
Backend: QtWebEngine 5.15.5, based on Chromium 87.0.4280.144
Qt: 5.15.2

CPython: 3.9.6
PyQt: 5.15.4

sip: 6.1.0.dev2104271705
colorama: 0.4.4
jinja2: 3.0.1
pygments: 2.9.0
yaml: 5.4.1
adblock: no
PyQt5.QtWebEngineWidgets: yes
PyQt5.QtWebEngine: 5.15.4
PyQt5.QtWebKitWidgets: yes
pdf.js: 2.9.359 (/usr/share/pdf.js/build/pdf.js)
sqlite: 3.36.0
QtNetwork SSL: OpenSSL 1.1.1k  25 Mar 2021

Style: QFusionStyle
Platform plugin: wayland
OpenGL: AMD, 4.6 (Compatibility Profile) Mesa 21.1.4
Platform: Linux-5.12.15-arch1-1-x86_64-with-glibc2.33, 64bit
Linux distribution: Arch Linux (arch)
Frozen: False
Imported from /usr/lib/python3.9/site-packages/qutebrowser
Using Python from /usr/bin/python3
Qt library executable path: /usr/lib/qt/libexec, data path: /usr/share/qt
```

**Does the bug happen if you start with `--temp-basedir`?**:
Yes, but _only if I run `:adblock-update` before attempting to reproduce_.

**Description**
In the [Amazon Web Services web console](https://console.aws.amazon.com/), there is a search bar at the top of the page that allows you to search AWS services and documentation. Up until recently, this search bar has worked okay. However, on updating to qutebrowser v2.3.0, it stopped working. I believe it is linked specifically to this change:

> The hosts-based adblocker (using `content.blocking.hosts.lists`) now also blocks all requests to any subdomains of blocked hosts.

Disabling adblocking with `:set content.blocking.enabled false` allows the search bar to work again.

Using dev tools, you can see requests being blocked to https://concierge.analytics.console.aws.a2z.com/search. The blocklist at https://github.com/StevenBlack/hosts contains an entry for analytics.console.aws.a2z.com, but _not_ for concierge.analytics.console.aws.a2z.com, which leads me to conclude that the blocked functionality is a result of the aforementioned change.

Perhaps a toggle to turn this subdomain blocking functionality off would be appropriate? I did not notice such a parameter mentioned in the changelog or when looking at parameters under `content.blocking`.

**How to reproduce**
1. Run `qutebrowser --temp-basedir`.
2. Update your adblock list using `:adblock-update`.
3. Navigate to https://console.aws.amazon.com/
4. Log in with a valid AWS account. Creating an AWS account is free.
5. In the search bar at the top of the page, attempt to search (any search string is OK). Notice the message "Unified search service is unavailable. Try your search again later."
6. Run `:set content.blocking.enabled false`.
7. Refresh the page.
8. In the search bar at the top, attempt to search (any search string is OK). Notice that the search functionality now works.



Discussion:
cc @4nd3r - didn't take too long until we found a site which breaks :rofl: 

@jkoelndorfer as a workaround, I think you should be able to add `concierge.analytics.console.aws.a2z.com` to `content.blocking.whitelist` - but I think it's probably a good idea to add an option for this regardless.

Indeed there isn't one at the moment, because we thought this should be mostly harmless. @4nd3r would you be interested in opening a PR to add one?
It's "whitelist that one domain" vs "make hostblocking worse for the rest". I would go with first solution, because I still want to have implicit subdomain blocking for everything else, but same time I fully support user choice. Looking at settings list, something like `content.blocking.hosts.block_subdomains` defaulting to `True` then?
Sounds good to me!
Ok, you can assign this issue to me and I'll start with it as soon as possible.

Probably this week or even today evening, but no promises given 🌞 
No hurry, I'll be quite busy the next couple of days/weeks too most likely.
@The-Compiler: Adding `concierge.analytics.console.aws.a2z.com` to `content.blocking.whitelist` does solve the issue.

Thank you for a great browser!

Resolving patch:
diff --git a/qutebrowser/components/hostblock.py b/qutebrowser/components/hostblock.py
--- a/qutebrowser/components/hostblock.py
+++ b/qutebrowser/components/hostblock.py
@@ -132,10 +132,16 @@ def _is_blocked(self, request_url: QUrl, first_party_url: QUrl = None) -> bool:
 
         host = request_url.host()
 
-        return any(
-            hostname in self._blocked_hosts or hostname in self._config_blocked_hosts
-            for hostname in urlutils.widened_hostnames(host)
-        )
+        if config.get("content.blocking.hosts.block_subdomains"):
+            return any(
+                hostname in self._blocked_hosts
+                or hostname in self._config_blocked_hosts
+                for hostname in urlutils.widened_hostnames(host)
+            )
+        else:
+            return (
+                host in self._blocked_hosts or host in self._config_blocked_hosts
+            )
 
     def filter_request(self, info: interceptor.Request) -> None:
         """Block the given request if necessary."""


Test patch:
diff --git a/tests/unit/components/test_hostblock.py b/tests/unit/components/test_hostblock.py
--- a/tests/unit/components/test_hostblock.py
+++ b/tests/unit/components/test_hostblock.py
@@ -565,9 +565,13 @@ def test_adblock_benchmark(data_tmpdir, benchmark, host_blocker_factory):
     benchmark(lambda: blocker._is_blocked(url))
 
 
-def test_subdomain_blocking(config_stub, host_blocker_factory):
+@pytest.mark.parametrize("block_subdomains", [True, False])
+def test_subdomain_blocking(config_stub, host_blocker_factory, block_subdomains):
     config_stub.val.content.blocking.method = "hosts"
     config_stub.val.content.blocking.hosts.lists = None
+    config_stub.val.content.blocking.hosts.block_subdomains = block_subdomains
     host_blocker = host_blocker_factory()
     host_blocker._blocked_hosts.add("example.com")
-    assert host_blocker._is_blocked(QUrl("https://subdomain.example.com"))
+    is_blocked = host_blocker._is_blocked(QUrl("https://subdomain.example.com"))
+    # block_subdomains is also expected result of is_blocked
+    assert is_blocked == block_subdomains