Repository: searx/searx

Issue:
[fix] url_for(..., _external=True) in templates
## What does this PR do?

Fix `webapp.url_for_theme` which is used in templates by `url_for(..., _external=True)`

## Why is this change important?

The `url_for` function in the template context is not the one from Flask, it is the one from `webapp`.  The `webapp.url_for_theme` is different from its namesake of Flask and has it quirks, when called with argument `_external=True`.

https://github.com/searx/searx/blob/9292571304376c2c463fb26ee0db89298bb00e6c/searx/webapp.py#L395-L396

The `webapp.url_for_theme` can't handle absolute URLs since it pokes a leading '/', here is the snippet of the old code::

https://github.com/searx/searx/blob/9292571304376c2c463fb26ee0db89298bb00e6c/searx/webapp.py#L311-L315

Next drawback of (Flask's) `_external=True` is, that it will not return the HTTP scheme when searx (the Flask app) listens on http and is proxied by a https server.

To get the right scheme `HTTP_X_SCHEME` is needed by Flask (werkzeug).  Since this is not provided in every environment (e.g. behind Apache mod_wsgi or the HTTP header is not fully set for some other reasons) it is recommended to get *script_name*, *server* and *scheme* from the configured `base_url`.  If `base_url` is specified, then these values from are given preference over any Flask's generics.

  The issue is discussed here https://github.com/searx/searx/discussions/2639

BTW this patch normalize to use `url_for` in the `opensearch.xml` and drop the need of `host` and `urljoin` in template's context.

## How to test this PR locally?

To test the issue discussed in https://github.com/searx/searx/discussions/2639 you will need a proxied searx instance with a https proxy (where `HTTP_X_SCHEME` is not set).  I tested it on my instance at https://darmarit.org/searx

----

Prerequisite-of: https://github.com/searx/searx/pull/2716



Discussion:
Can we prioritize this PR? 
@return42 can you try the starlette branch in your current configuration / setup: https://github.com/dalf/searx/tree/starlette and check if the issues are fixed (http instead of https for example) ?

The branch is not up to date but:
* the implementation of `url_for` is different:
  * https://github.com/dalf/searx/blob/e8bf00132f093eab9dba12d1de1ba79bdb8a0961/searx/web/templates.py#L88
  * https://github.com/encode/starlette/blob/1222e78a15336162d190060f76c8ccee4b011d6b/starlette/templating.py#L56-L59
* there is no [reverse proxy path fix](https://github.com/searx/searx/blob/ae122ea943f77600fd97556503c483dcd92e1e63/searx/webapp.py#L1113-L1147) ( https://github.com/wilbertom/flask-reverse-proxy )
>  can you try the starlette branch in your current configuration

I tried, but  the branch is about 476 commits behind master and I did not have luck to get it running in my darmarit.org/searx setup.

The patch `url_for_theme` function ..

https://github.com/dalf/searx/blob/e8bf00132f093eab9dba12d1de1ba79bdb8a0961/searx/web/templates.py#L88-L94

also drops the `del values['_external']` so I think this patch addresses the same problem I try to solve.  I would vote for my solution, where the _usage_ of the `_external` argument is dropped from the templates .. I think it is a better compared to drop arguments from an argument list.  

IMO the usage of the  the _usage_ of the `_external` argument and this usage should be fixed.

> there is no reverse proxy path fix ( https://github.com/wilbertom/flask-reverse-proxy )

Can't say anything about proxy path fix (haven't had time to analyse) and the http / https problem, but I think this problem is not related to this PR ... [The `webapp.url_for_theme` can't handle absolute (external) URLs since it pokes a leading `/`](https://github.com/searx/searx/pull/2656#issue-592577306)

> Next drawback of (Flask's) `_external=True` is, that it will not return the https scheme when searx (the Flask app) listens on http and is proxied by a https server.

this might be related to the proxy-path-fix .. but I think we should not use the `_external=True` at all, at least not in a way we use it ATM, with poking a  leading `/` in the `url_for_theme`  .. external URLs always have a leading `h` from `http...` :-)

Reading https://flask.palletsprojects.com/en/1.1.x/api/?highlight=url_for#flask.url_for
I discover that there are 3 config variables:
* `APPLICATION_ROOT`
* `SERVER_NAME`  (note: if not set, Flask uses the value from the `Host` header of the request.)
* `PREFERRED_URL_SCHEME`: (note: Flask does not seems to use the `X-Forwarded-Proto` header.)

So here a patch based on master :
```diff
diff --git a/searx/webapp.py b/searx/webapp.py
index b4466c71..ead475bd 100755
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -122,6 +122,13 @@ app.jinja_env.lstrip_blocks = True
 app.jinja_env.add_extension('jinja2.ext.loopcontrols')  # pylint: disable=no-member
 app.secret_key = settings['server']['secret_key']
 
+if settings['server']['base_url']:
+	base_url = urlparse(settings['server']['base_url'])
+	app.config['PREFERRED_URL_SCHEME'] = base_url.scheme
+	app.config['SERVER_NAME'] = base_url.netloc
+	app.config['APPLICATION_ROOT'] = base_url.path
+
+
 # see https://flask.palletsprojects.com/en/1.1.x/cli/
 # True if "FLASK_APP=searx/webapp.py FLASK_ENV=development flask run"
 flask_run_development = \
@@ -270,14 +277,7 @@ def extract_domain(url):
 
 
 def get_base_url():
-    if settings['server']['base_url']:
-        hostname = settings['server']['base_url']
-    else:
-        scheme = 'http'
-        if request.is_secure:
-            scheme = 'https'
-        hostname = url_for('index', _external=True, _scheme=scheme)
-    return hostname
+    return url_for('index', _external=True)
 
 
 def get_current_theme_name(override=None):
@@ -309,12 +309,7 @@ def url_for_theme(endpoint, override_theme=None, **values):
         filename_with_theme = "themes/{}/{}".format(theme_name, values['filename'])
         if filename_with_theme in static_files:
             values['filename'] = filename_with_theme
-    url = url_for(endpoint, **values)
-    if settings['server']['base_url']:
-        if url.startswith('/'):
-            url = url[1:]
-        url = urljoin(settings['server']['base_url'], url)
-    return url
+    return url_for(endpoint, **values)
 
 
 def proxify(url):
@@ -824,7 +819,7 @@ def preferences():
 
     # save preferences
     if request.method == 'POST':
-        resp = make_response(redirect(urljoin(settings['server']['base_url'], url_for('index'))))
+        resp = make_response(url_for('index', _external=True))
         try:
             request.preferences.parse_form(request.form)
         except ValidationException:
@@ -1038,7 +1033,7 @@ def favicon():
 
 @app.route('/clear_cookies')
 def clear_cookies():
-    resp = make_response(redirect(urljoin(settings['server']['base_url'], url_for('index'))))
+    resp = make_response(redirect(url_for('index', _external=True)))
     for cookie_name in request.cookies:
         resp.delete_cookie(cookie_name)
     return resp
```

Does it fix the issues?

If not, is it possible to keep one function to create the external URL?

```python
def url_for_external(relative_url):
    # TODO check there is no double / (one at the end of base_url, one at the beginning of relative_url)   
    return settings['server']['base_url'] + relative_url
```

So it is easier to adjust the URL creation in one place, and it avoids mistakes. For example should a `/` by added just after `{{base_url}}` ?

I'm sorry for the late review.
> I'm sorry for the late review.

Hey, no problem .. you released v1 which was a lot of work!

> If not, is it possible to keep one function to create the external URL?

I think this is the best solution.

> Does it fix the issues?

I will give it a try .. and come back.


> So here a patch based on master

@dalf thanks for the patch.  It was not the completion solution but points me in the right direction.  Instead of setting `app.conf` I patched the `ReverseProxyPathFix`, see commit message  (or my updated https://github.com/searx/searx/pull/2656#issue-592577306)

Why is it `X-Forwarded-Server` and not [`X-Forwarded-Host`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Host) ?
* The [nginx documentation](https://searx.github.io/searx/admin/installation-nginx.html?highlight=forwarded) makes a reference to `X-Forwarded-Host`.
* There is no `X-Forwarded-Host` for the [Apache documentation](https://searx.github.io/searx/admin/installation-apache.html), but a reference to [`ProxyPreserveHost`](https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#ProxyPreserveHost).

(I was not aware of `X-Forwarded-Server`, I found this https://stackoverflow.com/questions/43689625/x-forwarded-host-vs-x-forwarded-server I'm not sure if it is revelant or not )

----

Side note about `X-Scheme` (the value is already in the master branch):
* The [nginx documentation](https://searx.github.io/searx/admin/installation-nginx.html?highlight=forwarded) makes a reference to it.
* Not the [Apache documentation](https://searx.github.io/searx/admin/installation-apache.html) documentation.
* [`X-Forwarded-Proto`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto) seems more standard, but `X-Scheme` is fine too.

> Why is it X-Forwarded-Server and not X-Forwarded-Host ?

fixed it to `..-Host`

Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -1133,11 +1133,14 @@ def __init__(self, app):
 
             base_url = urlparse(settings['server']['base_url'])
             self.script_name = base_url.path
+            if self.script_name.endswith('/'):
+                # remove trailing slash to avoid infinite redirect on the index
+                # see https://github.com/searx/searx/issues/2729
+                self.script_name = self.script_name[:-1]
             self.scheme = base_url.scheme
             self.server = base_url.netloc
 
     def __call__(self, environ, start_response):
-
         script_name = self.script_name or environ.get('HTTP_X_SCRIPT_NAME', '')
         if script_name:
             environ['SCRIPT_NAME'] = script_name


Test patch: