Repository: qutebrowser/qutebrowser

Issue:
Reintroduce script for updating user agents
The [completions](https://github.com/qutebrowser/qutebrowser/blob/v1.14.0/qutebrowser/config/configdata.yml#L576-L586) for the `content.headers.user_agent` setting needs to be updated periodically to keep the suggested user agents recent.

We used to have a [script](https://github.com/qutebrowser/qutebrowser/blob/v1.8.3/scripts/dev/ua_fetch.py) to generate those entries, but it was removed in 5e64aae70be25e8871846af15ff92cfce7a816d9 because the underlying [website](https://techblog.willshouse.com/2012/01/03/most-common-user-agents/) added cloudflare protection so it's not possible to get the UAs automatically anymore.

However, it looks like there's a project called [top-user-agents](https://github.com/Kikobeats/top-user-agents) which periodically updates a [JSON file](https://github.com/Kikobeats/top-user-agents/blob/master/index.json) with the same information in a machine-readable format.

With that it should be possible to add a new script to `scripts/dev/` again, which does something like this:

- Use `requests` to fetch this JSON file from GitHub
- Get the entries using Python's `JSON` module
- Parse those user agents, perhaps using [`websettings.UserAgent.parse`](https://github.com/qutebrowser/qutebrowser/blob/v1.14.0/qutebrowser/config/websettings.py#L39-L78) from qutebrowser.
- Discard any user agent which is not Chrome (anything that doesn't contain `Chrome/...`; or anything that contains `Edg/...` or `OPR/...`)
- For each of Windows/Mac/Linux, find the user agent with the newest Chrome version and print it.

In a later step, that script could then be improved to automatically update `configdata.yml` instead of printing the user agents, and finally, this could be integrated with [GitHub Actions](https://github.com/features/actions) to automatically send a pull request once a month or so to update the values in the repository (or instead it could be added to the release automation).


Discussion:

Resolving patch:
diff --git a/scripts/dev/ua_fetch.py b/scripts/dev/ua_fetch.py
new file mode 100644
--- /dev/null
+++ b/scripts/dev/ua_fetch.py
@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+# vim: ft=python fileencoding=utf-8 sts=4 sw=4 et:
+
+"""\
+Fetch and print the most common user agents.
+
+This script fetches the most common user agents according to
+https://github.com/Kikobeats/top-user-agents, and prints the most recent
+Chrome user agent for Windows, macOS and Linux.
+"""
+
+import math
+import sys
+import textwrap
+
+import requests
+import qutebrowser.config.websettings
+
+
+def version(ua):
+    """Comparable version of a user agent."""
+    return tuple(int(v) for v in ua.upstream_browser_version.split('.')[:2])
+
+
+def wrap(ini, sub, string):
+    return textwrap.wrap(string, width=80, initial_indent=ini, subsequent_indent=sub)
+
+
+response = requests.get('https://raw.githubusercontent.com/Kikobeats/top-user-agents/master/index.json')
+
+if response.status_code != 200:
+    print('Unable to fetch the user agent index', file=sys.stderr)
+    sys.exit(1)
+
+ua_checks = {
+    'Win10': lambda ua: ua.os_info.startswith('Windows NT'),
+    'macOS': lambda ua: ua.os_info.startswith('Macintosh'),
+    'Linux': lambda ua: ua.os_info.startswith('X11'),
+}
+
+ua_strings = {}
+ua_versions = {}
+ua_names = {}
+
+for ua_string in reversed(response.json()):
+    # reversed to prefer more common versions
+
+    # Filter out browsers that are not Chrome-based
+    parts = ua_string.split()
+    if not any(part.startswith("Chrome/") for part in parts):
+        continue
+    if any(part.startswith("OPR/") or part.startswith("Edg/") for part in parts):
+        continue
+
+    user_agent = qutebrowser.config.websettings.UserAgent.parse(ua_string)
+
+    # check which os_string conditions are met and select the most recent version
+    for key, check in ua_checks.items():
+        if check(user_agent):
+            v = version(user_agent)
+            if v >= ua_versions.get(key, (-math.inf,)):
+                ua_versions[key] = v
+                ua_strings[key] = ua_string
+                ua_names[key] = f'Chrome {v[0]} {key}'
+
+for key, ua_string in ua_strings.items():
+    for line in wrap("      - - ", "          ", ua_string):
+        print(line)
+    for line in wrap("        - ", "          ", ua_names[key]):
+        print(line)


Test patch: