Repository: qutebrowser/qutebrowser

Issue:
Make it possible to not send "x-do-not-track" header
When `content.headers.do_not_track` is enabled, Qutebrowser will append two additional headers: `Dnt: 1` and `X-Do-Not-Track: 1`. When I disable that option, these headers are also sent, but with the value of `0`.
This causes problems with Android Messages site https://messages.android.com/ - and most likely with other sites which use strict CORS policy for headers.
What happens now:
1. `messages.android.com` uses `fetch` to send a cross-domain request to `https://instantmessaging-pa.googleapis.com/$rpc/google.internal.communications.instantmessaging.v1.Pairing/RegisterPhoneRelay`
2. `fetch` checks if there are some "non-standard" headers in the request. `Dnt` is considered a standard header, while `X-Do-Not-Track` is non-standard.
3. `fetch` sends a pre-flight `OPTIONS` request to ask if these non-standard headers are allowed: `access-control-request-headers: content-type,x-do-not-track,x-goog-api-key,x-user-agent`
4. Google's server doesn't allow the `x-do-not-track` header which it does not know about
5. `fetch` returns an error
6. Site doesn't work.

Obviously this issue is not specific to just this service.

How to resolve it:
1. Somehow patch `fetch` API to not consider `X-Do-Not-Track` header as non-standard.
2. Only send `DNT` header (like `Chromium` does)
3. Make it configurable whether to use `X-Do-Not-Track` header or not
4. Don't send any DNT headers if `content.headers.do_not_track` is disabled, instead of sending them with zero values.


Discussion:
I think both 2) and 4) are a good idea - I don't think `X-Do-Not-Track` should still be sent nowadays, but the setting also should have three possible values.
Looks like `content.headers.do_not_track` option is intended to accept `none` as a vaule, but in fact I could not find a way to set such value via user interface. If I try to then I get error `Invalid value 'null' - must be a boolean!`. Despite in `configdata.yml` this option has `type.none_ok: true` set.
Okay, tried to set it to empty string `:set content.headers.do_not_track ''` and it worked. But it is not obvious and seemingly not documented.


> Okay, tried to set it to empty string :set content.headers.do_not_track '' and it worked. But it is not obvious and seemingly not documented.

Hmm, maybe that shouldn't work, being able to set boolean values to empty
string seems a little weird to me. I think that setting should be migrated to
a string with a third option though, so this behavior can be properly changed
without hacks (as discussed before). Should be pretty easy to do, I think.


Resolving patch:
diff --git a/qutebrowser/browser/shared.py b/qutebrowser/browser/shared.py
--- a/qutebrowser/browser/shared.py
+++ b/qutebrowser/browser/shared.py
@@ -42,7 +42,6 @@ def custom_headers(url):
     if dnt_config is not None:
         dnt = b'1' if dnt_config else b'0'
         headers[b'DNT'] = dnt
-        headers[b'X-Do-Not-Track'] = dnt
 
     conf_headers = config.instance.get('content.headers.custom', url=url)
     for header, value in conf_headers.items():


Test patch:
diff --git a/tests/end2end/features/misc.feature b/tests/end2end/features/misc.feature
--- a/tests/end2end/features/misc.feature
+++ b/tests/end2end/features/misc.feature
@@ -331,19 +331,16 @@ Feature: Various utility commands.
         When I set content.headers.do_not_track to true
         And I open headers
         Then the header Dnt should be set to 1
-        And the header X-Do-Not-Track should be set to 1
 
     Scenario: DNT header (off)
         When I set content.headers.do_not_track to false
         And I open headers
         Then the header Dnt should be set to 0
-        And the header X-Do-Not-Track should be set to 0
 
     Scenario: DNT header (unset)
         When I set content.headers.do_not_track to <empty>
         And I open headers
         Then the header Dnt should be set to <unset>
-        And the header X-Do-Not-Track should be set to <unset>
 
     Scenario: Accept-Language header
         When I set content.headers.accept_language to en,de
diff --git a/tests/unit/browser/test_shared.py b/tests/unit/browser/test_shared.py
--- a/tests/unit/browser/test_shared.py
+++ b/tests/unit/browser/test_shared.py
@@ -26,18 +26,15 @@
 
 @pytest.mark.parametrize('dnt, accept_language, custom_headers, expected', [
     # DNT
-    (True, None, {}, {b'DNT': b'1', b'X-Do-Not-Track': b'1'}),
-    (False, None, {}, {b'DNT': b'0', b'X-Do-Not-Track': b'0'}),
+    (True, None, {}, {b'DNT': b'1'}),
+    (False, None, {}, {b'DNT': b'0'}),
     (None, None, {}, {}),
     # Accept-Language
-    (False, 'de, en', {}, {b'DNT': b'0', b'X-Do-Not-Track': b'0',
-                           b'Accept-Language': b'de, en'}),
+    (False, 'de, en', {}, {b'DNT': b'0', b'Accept-Language': b'de, en'}),
     # Custom headers
-    (False, None, {'X-Qute': 'yes'}, {b'DNT': b'0', b'X-Do-Not-Track': b'0',
-                                      b'X-Qute': b'yes'}),
+    (False, None, {'X-Qute': 'yes'}, {b'DNT': b'0', b'X-Qute': b'yes'}),
     # Mixed
     (False, 'de, en', {'X-Qute': 'yes'}, {b'DNT': b'0',
-                                          b'X-Do-Not-Track': b'0',
                                           b'Accept-Language': b'de, en',
                                           b'X-Qute': b'yes'}),
 ])