Repository: searx/searx

Issue:
HTTP headers
searx includes contents from others websites. The contents are escaped, trimed and all, but there is always a risk. The security could be improve using HTTP headers. 

Some headers that could be added : 

`X-Frame-Options: SAMEORIGIN`
when there is the image proxy (break the included videos, in this case a more precise).
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

`X-Content-Security-Policy "default-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self' 'unsafe-inline'; connect-src 'self';"`
to be adjust when there are images, videos
https://content-security-policy.com/
https://report-uri.io/

`X-Content-Type-Options: nosniff`
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

`X-XSS-Protection "1; mode=block"`
https://blogs.msdn.microsoft.com/ieinternals/2011/01/31/controlling-the-xss-filter/

`X-Download-Options: noopen`
https://msdn.microsoft.com/fr-fr/library/jj542450(v=vs.85).aspx#noOpenDirective

more or less out of topic
`X-Robots-Tag: noindex, nofollow`
http://robots-txt.com/x-robots-tag/
avoid to be indexed and then spam (useless for that purpose ?)

HSTS header ( see https://github.com/asciimoo/searx/issues/62 )
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
https://hstspreload.appspot.com/

Should nginx / apache send the headers or should the python code should send them ? in case of X-Content-Security-Policy and X-Frame-Options the values can change according to the search options.

HTTP headers
searx includes contents from others websites. The contents are escaped, trimed and all, but there is always a risk. The security could be improve using HTTP headers. 

Some headers that could be added : 

`X-Frame-Options: SAMEORIGIN`
when there is the image proxy (break the included videos, in this case a more precise).
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

`X-Content-Security-Policy "default-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; script-src 'self' 'unsafe-inline'; connect-src 'self';"`
to be adjust when there are images, videos
https://content-security-policy.com/
https://report-uri.io/

`X-Content-Type-Options: nosniff`
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

`X-XSS-Protection "1; mode=block"`
https://blogs.msdn.microsoft.com/ieinternals/2011/01/31/controlling-the-xss-filter/

`X-Download-Options: noopen`
https://msdn.microsoft.com/fr-fr/library/jj542450(v=vs.85).aspx#noOpenDirective

more or less out of topic
`X-Robots-Tag: noindex, nofollow`
http://robots-txt.com/x-robots-tag/
avoid to be indexed and then spam (useless for that purpose ?)

HSTS header ( see https://github.com/asciimoo/searx/issues/62 )
https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security
https://hstspreload.appspot.com/

Should nginx / apache send the headers or should the python code should send them ? in case of X-Content-Security-Policy and X-Frame-Options the values can change according to the search options.



Discussion:
IMHO it would be better to have the web servers create the headers if you want to have additional headers (like key pinning stuff).  But because Apache/mod_uwsgi does not work with header add I hard code my headers in webapp.py by now which is not very easy because I need to edit webapp.py with every git pull. Is there an mod_proxy_uwsgi description for searx? mod_proxy_uwsgi makes web server side headers possible.

A solution about hard coded headers in webapp is settings.yml. But it would be messy to have header inside settings.yml and inside the HTTP server.
I agree it should be in the web server, so it is related to the installation.

after reverse proxying my uwsgi <-> apache connection by http all my headers are in apache now. So updating is easier now because there is no webapp.py patching anymore. Headers in apache is also good if you are intercepting by filtron (if it works) because then you do not have problems with loosing headers.

It would be good to have decent defaults for http headers, but I'm not sure if the searx webapp is the right place for these headers. What if someone wants to change it for some reason? Should we add a configuration section for HTTP headers? Also, there is a possibility that adding new default headers may break someones existing configuration (CSP).

As a backward compatible solution, I'd suggest to do not turn these headers on by default, but allow admins to turn it on from `settings.yml`.

What do you think?
I'm afraid that off by default means "it doesn't exist" most of the time: if these headers would break something we would have feedback, but there are mostly seat belts (except the Cache-Control), so most of the time even turn off everything will be look good (same for a wrong TLS configuration).

But as you said, if some headers are enabled by default, it may conflict with some existing configurations.

The only way I see is a two phase activation:
* add some headers, disabled by default, display a warning (where ?) that they will be enabled at one point in the future (to be defined in relative or absolute time).
* enabled them by default later (in a new release ?).

Can be related to https://github.com/searx/searx-instances/issues/70 :
* searx-stats2 can check the headers (if there are multiple values for the same header, or if there is some missing headers according to the good practice).
* we can provide an API to check an existing instance : a script can call this API (in the installation script for example). But there are a lot of external tutorials which described how to install searx : they won't be updated.

Does it worth it ? I'm not sure where to put the cursor.

-----

Looking at https://github.com/searx/searx-docker/blob/master/Caddyfile (not updated since last year, may be it requires some updates ? may be some values are broken ?).

#### CSP
documentation: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

value in searx-docker: https://github.com/searx/searx-docker/blob/7ad4d4903d0ce41de4094d5fe0fecda156b41eaa/Caddyfile#L83 

The value of this header depends on the searx code (tile.openstreetmap.org hostname etc...)

I think a default value would be good : currently we can't change this value without issue.

#### X-XSS-Protection
The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection

To keep the backward compatibility for some old browsers, we can keep this header.

#### X-Frame-Options
The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

To keep the backward compatibility for some old browsers, we can keep this header.

#### X-Content-Type-Options

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

Not deprecated, seems good whatever the usage context

#### Feature-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Feature_Policy

It is a seat belt: searx doesn't use these features, if for some reason there is an XSS, at least this header will block usage of these features.

Here, my take is HTTP/2 allows to send a reference to some previous headers so it doesn't cost anything to add this header.

#### Referrer-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

Seems good, it is redundant with the templates : 
https://github.com/searx/searx/blob/db703a0283ee169381aeea97c678e666ae508348/searx/templates/oscar/base.html#L10

#### X-Robots-Tag

According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers :
The header is effectively equivalent to `<meta name="robots" content="...">.`

#### Cache-Control

Related to 
* https://github.com/searx/searx/pull/2242#issuecomment-706522242
* https://github.com/searx/searx/issues/2231 (since the page is not cached, the back button reloads the page).

I think a default value would be good at least for the static files ?

I'm not sure what to do for the index and preferences page ( searx-docker set the value of "no-cache, no-store", but some instances doesn't set this value.

Good points. What about adding the headers that most probably can't cause backward incompatibility and creating a warning for adding the rest in the future?
Except Cache-Control for the /index and /preferences pages, I would declare all these headers in settings.yml , section `server.headers` for example. 

The main issue is the duplicate headers: the reverse proxy most probably will add or replace some headers too. Replace should not be a problem (but it produces an expect configuration), add may create a wrong / unsecure configuration.

* Caddy replace the headers according to https://caddyserver.com/docs/caddyfile/directives/header#header
* nginx: I don't know : http://nginx.org/en/docs/http/ngx_http_headers_module.html
* Apache httpd: seems okay according to https://httpd.apache.org/docs/current/en/mod/mod_headers.html (`Header set`  directive).

Adding headers in settings.yml is related to https://github.com/searx/searx/issues/1209 : settings.yml won't be updated blindly if it contains some specific configuration (like the secret key, the engine tokens, etc...)

IMHO it would be better to have the web servers create the headers if you want to have additional headers (like key pinning stuff).  But because Apache/mod_uwsgi does not work with header add I hard code my headers in webapp.py by now which is not very easy because I need to edit webapp.py with every git pull. Is there an mod_proxy_uwsgi description for searx? mod_proxy_uwsgi makes web server side headers possible.

A solution about hard coded headers in webapp is settings.yml. But it would be messy to have header inside settings.yml and inside the HTTP server.
I agree it should be in the web server, so it is related to the installation.

after reverse proxying my uwsgi <-> apache connection by http all my headers are in apache now. So updating is easier now because there is no webapp.py patching anymore. Headers in apache is also good if you are intercepting by filtron (if it works) because then you do not have problems with loosing headers.

It would be good to have decent defaults for http headers, but I'm not sure if the searx webapp is the right place for these headers. What if someone wants to change it for some reason? Should we add a configuration section for HTTP headers? Also, there is a possibility that adding new default headers may break someones existing configuration (CSP).

As a backward compatible solution, I'd suggest to do not turn these headers on by default, but allow admins to turn it on from `settings.yml`.

What do you think?
I'm afraid that off by default means "it doesn't exist" most of the time: if these headers would break something we would have feedback, but there are mostly seat belts (except the Cache-Control), so most of the time even turn off everything will be look good (same for a wrong TLS configuration).

But as you said, if some headers are enabled by default, it may conflict with some existing configurations.

The only way I see is a two phase activation:
* add some headers, disabled by default, display a warning (where ?) that they will be enabled at one point in the future (to be defined in relative or absolute time).
* enabled them by default later (in a new release ?).

Can be related to https://github.com/searx/searx-instances/issues/70 :
* searx-stats2 can check the headers (if there are multiple values for the same header, or if there is some missing headers according to the good practice).
* we can provide an API to check an existing instance : a script can call this API (in the installation script for example). But there are a lot of external tutorials which described how to install searx : they won't be updated.

Does it worth it ? I'm not sure where to put the cursor.

-----

Looking at https://github.com/searx/searx-docker/blob/master/Caddyfile (not updated since last year, may be it requires some updates ? may be some values are broken ?).

#### CSP
documentation: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy

value in searx-docker: https://github.com/searx/searx-docker/blob/7ad4d4903d0ce41de4094d5fe0fecda156b41eaa/Caddyfile#L83 

The value of this header depends on the searx code (tile.openstreetmap.org hostname etc...)

I think a default value would be good : currently we can't change this value without issue.

#### X-XSS-Protection
The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection

To keep the backward compatibility for some old browsers, we can keep this header.

#### X-Frame-Options
The header seems deprecated (replaced by CSP) : https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options

To keep the backward compatibility for some old browsers, we can keep this header.

#### X-Content-Type-Options

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options

Not deprecated, seems good whatever the usage context

#### Feature-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Feature_Policy

It is a seat belt: searx doesn't use these features, if for some reason there is an XSS, at least this header will block usage of these features.

Here, my take is HTTP/2 allows to send a reference to some previous headers so it doesn't cost anything to add this header.

#### Referrer-Policy

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

Seems good, it is redundant with the templates : 
https://github.com/searx/searx/blob/db703a0283ee169381aeea97c678e666ae508348/searx/templates/oscar/base.html#L10

#### X-Robots-Tag

According to https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers :
The header is effectively equivalent to `<meta name="robots" content="...">.`

#### Cache-Control

Related to 
* https://github.com/searx/searx/pull/2242#issuecomment-706522242
* https://github.com/searx/searx/issues/2231 (since the page is not cached, the back button reloads the page).

I think a default value would be good at least for the static files ?

I'm not sure what to do for the index and preferences page ( searx-docker set the value of "no-cache, no-store", but some instances doesn't set this value.

Good points. What about adding the headers that most probably can't cause backward incompatibility and creating a warning for adding the rest in the future?
Except Cache-Control for the /index and /preferences pages, I would declare all these headers in settings.yml , section `server.headers` for example. 

The main issue is the duplicate headers: the reverse proxy most probably will add or replace some headers too. Replace should not be a problem (but it produces an expect configuration), add may create a wrong / unsecure configuration.

* Caddy replace the headers according to https://caddyserver.com/docs/caddyfile/directives/header#header
* nginx: I don't know : http://nginx.org/en/docs/http/ngx_http_headers_module.html
* Apache httpd: seems okay according to https://httpd.apache.org/docs/current/en/mod/mod_headers.html (`Header set`  directive).

Adding headers in settings.yml is related to https://github.com/searx/searx/issues/1209 : settings.yml won't be updated blindly if it contains some specific configuration (like the secret key, the engine tokens, etc...)


Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -487,6 +487,16 @@ def pre_request():
             request.user_plugins.append(plugin)
 
 
+@app.after_request
+def add_default_headers(response):
+    # set default http headers
+    for header, value in settings['server'].get('default_http_headers', {}).items():
+        if header in response.headers:
+            continue
+        response.headers[header] = value
+    return response
+
+
 @app.after_request
 def post_request(response):
     total_time = time() - request.start_time


Test patch: