Repository: searxng/searxng

Issue:
[Feature] Mullvad Leta
**Working URL to the engine**
https://leta.mullvad.net/

**Why do you want to add this engine?**
Mullvad is a reputable VPN provider. If I could have my local instance also be a proxy to this search engine, that would reduce the number of devices I need connected to Mullvad to run to access this search feature.

**Features of this engine**
The blog post [here](https://leta.mullvad.net/faq) contains the features and FAQ. Their meta search engine uses the official paid Google API, so results are likely quite robust and not error prone to changes.

Technically Mullvad Leta is already a meta search engine on Google - I wonder if they use searx lol (probably not, they use NodeJS).

**How can SearXNG fetch the information from this engine?**
TBD - will likely need to scrape the front end

**Applicable category of this engine**
general

**Additional context**
You MUST have a Mullvad account to use this search engine. No account, no searches.

There is a limit on 100 non-cached searches (or credits) per valid Mullvad VPN account per 24 hours. You can search the cache without using credits.

**Notes**
I plan on working on this integration when I can, as it directly benefits myself! But anyone else can work on it too.


Discussion:
Additional and pretty important info:

> What is a cached search?#

> We store every search in a RAM based cache storage (Redis), which is removed after it reaches over 30 days in age.

> Cached searches are fetched from this storage, which means we return a result that can be from 0 to 30 days old. It may be the case that no other user has searched for something during the time that you search, which means you would be shown a stale result.


Also this:

>Each time you search for a phrase you use up 1 of the 100 credits.

>Each time you select next page you use up 1 of the 100 credits.

>If you select ”Only search in cache”, which is the default option, 0 credits are used.
1. multipart post request to https://leta.mullvad.net/login, `account_number=<account number without spaces>, q=<empty>, oc=<empty>` (presumably both can be unset but i haven't checked). Response is json `{"type":"redirect","status":302,"location":"/"}` (funnily http code is 200) and a set-cookie with `accessToken=...; Path=/; HttpOnly; Secure; SameSite=Strict`

2. https://leta.mullvad.net/__data.json should return a json with

- type="data"
- nodes= a list of objects with
  - type="data"
  - uses = some unknown object, can have no fields or have "url" set to 1, from reverse engineering the js it can have "dependencies" (array), "params" (array), "parent" (bool), "route" (bool), "url" (bool). I guess this is just some framework
  - data = an array, i've seen it set to
    - a single element set to null
    - first element = an object with connected=1, and the second element is just true
    - first element = an object with isLoggedIn=1, and the second element is just true

Either way, this json seems useless for searx, although it can be used to check isLoggedIn

3. send multipart request to https://leta.mullvad.net/, `q=<query>`, optional oc=on to only search cache (if oc is off it isnt present at all), optional `gl=<two lowercase letters specifying the region>` (it isnt present when doing a search from the landing page, but is set to an empty string when doing a search from the search results page without specifying a country)



Resolving patch:
diff --git a/searx/engines/mullvad_leta.py b/searx/engines/mullvad_leta.py
new file mode 100644
--- /dev/null
+++ b/searx/engines/mullvad_leta.py
@@ -0,0 +1,173 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+# lint: pylint
+
+"""This is the implementation of the Mullvad-Leta meta-search engine.
+
+This engine **REQUIRES** that searxng operate within a Mullvad VPN
+
+If using docker, consider using gluetun for easily connecting to the Mullvad
+
+- https://github.com/qdm12/gluetun
+
+Otherwise, follow instructions provided by Mullvad for enabling the VPN on Linux
+
+- https://mullvad.net/en/help/install-mullvad-app-linux
+
+.. hint::
+
+   The :py:obj:`EngineTraits` is empty by default.  Maintainers have to run
+   ``make data.traits`` (in the Mullvad VPN / :py:obj:`fetch_traits`) and rebase
+   the modified JSON file ``searx/data/engine_traits.json`` on every single
+   update of SearXNG!
+"""
+
+from typing import TYPE_CHECKING
+from httpx import Response
+from lxml import html
+from searx.enginelib.traits import EngineTraits
+from searx.locales import region_tag, get_official_locales
+from searx.utils import eval_xpath, extract_text, eval_xpath_list
+from searx.exceptions import SearxEngineResponseException
+
+if TYPE_CHECKING:
+    import logging
+
+    logger = logging.getLogger()
+
+traits: EngineTraits
+
+use_cache: bool = True  # non-cache use only has 100 searches per day!
+
+search_url = "https://leta.mullvad.net"
+
+# about
+about = {
+    "website": search_url,
+    "wikidata_id": 'Q47008412',  # the Mullvad id - not leta, but related
+    "official_api_documentation": 'https://leta.mullvad.net/faq',
+    "use_official_api": False,
+    "require_api_key": False,
+    "results": 'HTML',
+}
+
+# engine dependent config
+categories = ['general', 'web']
+paging = True
+max_page = 50
+time_range_support = True
+time_range_dict = {
+    "day": "d1",
+    "week": "w1",
+    "month": "m1",
+    "year": "y1",
+}
+
+
+def is_vpn_connected(dom: html.HtmlElement) -> bool:
+    """Returns true if the VPN is connected, False otherwise"""
+    connected_text = extract_text(eval_xpath(dom, '//main/div/p[1]'))
+    return connected_text != 'You are not connected to Mullvad VPN.'
+
+
+def assign_headers(headers: dict) -> dict:
+    """Assigns the headers to make a request to Mullvad Leta"""
+    headers['Accept'] = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"
+    headers['Content-Type'] = "application/x-www-form-urlencoded"
+    headers['Host'] = "leta.mullvad.net"
+    headers['Origin'] = "https://leta.mullvad.net"
+    return headers
+
+
+def request(query: str, params: dict):
+    country = traits.get_region(params.get('searxng_locale', 'all'), traits.all_locale)  # type: ignore
+
+    params['url'] = search_url
+    params['method'] = 'POST'
+    params['data'] = {
+        "q": query,
+        "gl": country if country is str else '',
+    }
+    # pylint: disable=undefined-variable
+    if use_cache:
+        params['data']['oc'] = "on"
+    # pylint: enable=undefined-variable
+
+    if params['time_range'] in time_range_dict:
+        params['dateRestrict'] = time_range_dict[params['time_range']]
+    else:
+        params['dateRestrict'] = ''
+
+    if params['pageno'] > 1:
+        #  Page 1 is n/a, Page 2 is 11, page 3 is 21, ...
+        params['data']['start'] = ''.join([str(params['pageno'] - 1), "1"])
+
+    if params['headers'] is None:
+        params['headers'] = {}
+
+    assign_headers(params['headers'])
+    return params
+
+
+def extract_result(dom_result: html.HtmlElement):
+    [a_elem, h3_elem, p_elem] = eval_xpath_list(dom_result, 'div/div/*')
+    return {
+        'url': extract_text(a_elem.text),
+        'title': extract_text(h3_elem),
+        'content': extract_text(p_elem),
+    }
+
+
+def response(resp: Response):
+    """Checks if connected to Mullvad VPN, then extracts the search results from
+    the DOM resp: requests response object"""
+
+    dom = html.fromstring(resp.text)
+    if not is_vpn_connected(dom):
+        raise SearxEngineResponseException('Not connected to Mullvad VPN')
+    search_results = eval_xpath(dom.body, '//main/div[2]/div')
+    return [extract_result(sr) for sr in search_results]
+
+
+def fetch_traits(engine_traits: EngineTraits):
+    """Fetch languages and regions from Mullvad-Leta
+
+    .. warning::
+
+        Fetching the engine traits also requires a Mullvad VPN connection. If
+        not connected, then an error message will print and no traits will be
+        updated.
+    """
+    # pylint: disable=import-outside-toplevel
+    # see https://github.com/searxng/searxng/issues/762
+    from searx.network import post as http_post
+
+    # pylint: enable=import-outside-toplevel
+    resp = http_post(search_url, headers=assign_headers({}))
+    if not isinstance(resp, Response):
+        print("ERROR: failed to get response from mullvad-leta. Are you connected to the VPN?")
+        return
+    if not resp.ok:
+        print("ERROR: response from mullvad-leta is not OK. Are you connected to the VPN?")
+        return
+    dom = html.fromstring(resp.text)
+    if not is_vpn_connected(dom):
+        print('ERROR: Not connected to Mullvad VPN')
+        return
+    # supported region codes
+    options = eval_xpath_list(dom.body, '//main/div/form/div[2]/div/select[1]/option')
+    if options is None or len(options) <= 0:
+        print('ERROR: could not find any results. Are you connected to the VPN?')
+    for x in options:
+        eng_country = x.get("value")
+
+        sxng_locales = get_official_locales(eng_country, engine_traits.languages.keys(), regional=True)
+
+        if not sxng_locales:
+            print(
+                "ERROR: can't map from Mullvad-Leta country %s (%s) to a babel region."
+                % (x.get('data-name'), eng_country)
+            )
+            continue
+
+        for sxng_locale in sxng_locales:
+            engine_traits.regions[region_tag(sxng_locale)] = eng_country


Test patch: