Repository: aio-libs/aiohttp

Issue:
Insecure algorithm for determining remote/scheme/host
## Long story short

`aiohttp.web.BaseRequest` has attributes [`scheme`](http://aiohttp.readthedocs.io/en/latest/web_reference.html#aiohttp.web.BaseRequest.scheme), [`host`](http://aiohttp.readthedocs.io/en/latest/web_reference.html#aiohttp.web.BaseRequest.host) and, since 2.3, [`remote`](http://aiohttp.readthedocs.io/en/latest/web_reference.html#aiohttp.web.BaseRequest.remote). Their values are determined by looking at request headers that can be manipulated by a remote user. Under most configurations, the user can set them to **any desired values**. This is especially bad for the `remote` attribute, because the user’s IP address is often relied upon for access controls.

(I’m reporting this in the open rather than privately, because `remote` is not yet released, while `scheme` and `host` do not seem like much of a security problem.)

## Expected behaviour

This involves the headers [`Forwarded`](https://tools.ietf.org/html/rfc7239) and `X-Forwarded-For`. The **last element** (comma-separated) in these headers can be trusted **as long as a trusted proxy is configured to append it**.

The same applies to the headers `X-Forwarded-Host` and `X-Forwarded-Proto`, with the exception that they are usually single values (to be replaced by the proxy), not comma-separated lists.

## Actual behaviour

aiohttp takes the **first element** from `Forwarded`/`X-Forwarded-For`, **without knowing which of these headers (if any) are controlled by a trusted proxy**.

In most deployments where aiohttp sits behind nginx, the `Forwarded` header is not controlled by the proxy. Nobody is aware that it needs to be controlled. It is not mentioned in the [example nginx configuration](http://aiohttp.readthedocs.io/en/latest/deployment.html#nginx-configuration) from aiohttp docs. In such deployments, an external user can trivially force `scheme`, `host` and `remote` to any desired values by sending a header like:

```
Forwarded: for=10.0.0.1;host=example.net;proto=https
```

In fact, it’s impossible to configure current versions of nginx to *correctly* append a `Forwarded` header (without writing some C or possibly Lua code).

But even if aiohttp was sitting behind a proxy that correctly controlled all of the involved headers, nothing would change, because the proxy would *append* a comma-separated element to the remote user’s `Forwarded` header, while aiohttp is looking at the *first* element — which is still controlled by the user.

## Steps to reproduce

Run this server program:

    from aiohttp import web
    async def handle(request):
        info = (request.scheme, request.host, request.remote)
        return web.Response(text=repr(info))
    app = web.Application()
    app.router.add_get('/', handle)
    web.run_app(app)

behind nginx with the following configuration (derived from the [example](http://aiohttp.readthedocs.io/en/latest/deployment.html#nginx-configuration)):

    daemon off;
    error_log stderr;
    pid /tmp/nginx1.pid;
    events {
    }
    http {
      server {
        listen 12345;
        server_name example.com;
        access_log /tmp/nginx1.access.log;
        location / {
          proxy_set_header Host $http_host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_redirect off;
          proxy_buffering off;
          proxy_pass http://localhost:8080;
        }
      }
    }

and send requests to it with curl:

    $ curl -s localhost:12345/ -H 'Forwarded: for=10.0.0.1;host=example.net;proto=https'
    ('https', 'example.net', '10.0.0.1')

## Your environment

aiohttp Git master, Python 3.5, Linux



Discussion:
Sorry, I don't follow what do you propose?
@asvetlov 

1. Remove `remote`.
2. Make `scheme` and `host` look at the last element of `Forwarded`, not the first.
3. Think about some sort of configuration options to indicate which headers can be trusted. Then `remote` can be restored (again, looking at the last element, not the first).

For example, see Gunicorn’s [`secure_scheme_headers`](http://docs.gunicorn.org/en/latest/settings.html#secure-scheme-headers) and [`forwarded_allow_ips`](http://docs.gunicorn.org/en/latest/settings.html#forwarded-allow-ips) settings. (Their default for `secure_scheme_headers` is too broad, but they have privately indicated an interest in changing it in a future version.)
Actually, rather than hardcoding “last element”, it might be better to provide, again, an option to choose it — like the `num_proxies` parameter to Werkzeug’s [`ProxyFix`](http://werkzeug.pocoo.org/docs/0.12/contrib/fixers/#werkzeug.contrib.fixers.ProxyFix) — for cases when aiohttp is not immediately behind the “front-end” proxy.
People will reimplement `remote` prop anyway, and most likely their implementation will be less safe than provided by aiohttp. Be could disable forwarded headers processing by default though.
Last element is not satisfactory in case of chain of two reverse proxies in front of  aiohttp server.

I like idea of additional opt-in application parameters for configuring corresponding header.
@asvetlov The parameter could combine headers and indices, like this:

    # Use the first element from Forwarded
    app.trusted_proxy_headers = {'Forwarded': 0}

    # Use the second-last element from X-Forwarded-For,
    # as well as any X-Forwarded-Proto
    app.trusted_proxy_headers = {'X-Forwarded-For': -2,
                                 'X-Forwarded-Proto': True}


Resolving patch:
diff --git a/aiohttp/web.py b/aiohttp/web.py
--- a/aiohttp/web.py
+++ b/aiohttp/web.py
@@ -50,7 +50,6 @@ def __init__(self, *,
                  middlewares=(),
                  handler_args=None,
                  client_max_size=1024**2,
-                 secure_proxy_ssl_header=None,
                  loop=None,
                  debug=...):
         if router is None:
@@ -60,13 +59,8 @@ def __init__(self, *,
         if loop is not None:
             warnings.warn("loop argument is deprecated", ResourceWarning)
 
-        if secure_proxy_ssl_header is not None:
-            warnings.warn(
-                "secure_proxy_ssl_header is deprecated", DeprecationWarning)
-
         self._debug = debug
         self._router = router
-        self._secure_proxy_ssl_header = secure_proxy_ssl_header
         self._loop = loop
         self._handler_args = handler_args
         self.logger = logger
@@ -235,8 +229,7 @@ def router(self):
     def middlewares(self):
         return self._middlewares
 
-    def make_handler(self, *, loop=None,
-                     secure_proxy_ssl_header=None, **kwargs):
+    def make_handler(self, *, loop=None, **kwargs):
         self._set_loop(loop)
         self.freeze()
 
@@ -245,8 +238,6 @@ def make_handler(self, *, loop=None,
             for k, v in self._handler_args.items():
                 kwargs[k] = v
 
-        if secure_proxy_ssl_header:
-            self._secure_proxy_ssl_header = secure_proxy_ssl_header
         return Server(self._handle, request_factory=self._make_request,
                       loop=self.loop, **kwargs)
 
@@ -279,7 +270,6 @@ def _make_request(self, message, payload, protocol, writer, task,
         return _cls(
             message, payload, protocol, writer, protocol._time_service, task,
             self._loop,
-            secure_proxy_ssl_header=self._secure_proxy_ssl_header,
             client_max_size=self._client_max_size)
 
     @asyncio.coroutine
diff --git a/aiohttp/web_request.py b/aiohttp/web_request.py
--- a/aiohttp/web_request.py
+++ b/aiohttp/web_request.py
@@ -3,6 +3,7 @@
 import datetime
 import json
 import re
+import socket
 import string
 import tempfile
 import types
@@ -62,8 +63,9 @@ class BaseRequest(collections.MutableMapping, HeadersMixin):
 
     def __init__(self, message, payload, protocol, writer, time_service, task,
                  loop,
-                 *, secure_proxy_ssl_header=None, client_max_size=1024**2,
-                 state=None):
+                 *, client_max_size=1024**2,
+                 state=None,
+                 scheme=None, host=None, remote=None):
         if state is None:
             state = {}
         self._message = message
@@ -79,7 +81,6 @@ def __init__(self, message, payload, protocol, writer, time_service, task,
         self._post = None
         self._read_bytes = None
 
-        self._secure_proxy_ssl_header = secure_proxy_ssl_header
         self._time_service = time_service
         self._state = state
         self._cache = {}
@@ -87,8 +88,13 @@ def __init__(self, message, payload, protocol, writer, time_service, task,
         self._client_max_size = client_max_size
         self._loop = loop
 
+        self._scheme = scheme
+        self._host = host
+        self._remote = remote
+
     def clone(self, *, method=sentinel, rel_url=sentinel,
-              headers=sentinel):
+              headers=sentinel, scheme=sentinel, host=sentinel,
+              remote=sentinel):
         """Clone itself with replacement some attributes.
 
         Creates and returns a new instance of Request object. If no parameters
@@ -115,6 +121,14 @@ def clone(self, *, method=sentinel, rel_url=sentinel,
 
         message = self._message._replace(**dct)
 
+        kwargs = {}
+        if scheme is not sentinel:
+            kwargs['scheme'] = scheme
+        if host is not sentinel:
+            kwargs['host'] = host
+        if remote is not sentinel:
+            kwargs['remote'] = remote
+
         return self.__class__(
             message,
             self._payload,
@@ -123,8 +137,8 @@ def clone(self, *, method=sentinel, rel_url=sentinel,
             self._time_service,
             self._task,
             self._loop,
-            secure_proxy_ssl_header=self._secure_proxy_ssl_header,
-            state=self._state.copy())
+            state=self._state.copy(),
+            **kwargs)
 
     @property
     def task(self):
@@ -175,15 +189,12 @@ def __iter__(self):
 
     @property
     def secure(self):
-        """A bool indicating if the request is handled with SSL or
-        'secure_proxy_ssl_header' is matching
-
-        """
-        return self.url.scheme == 'https'
+        """A bool indicating if the request is handled with SSL."""
+        return self.scheme == 'https'
 
     @reify
     def forwarded(self):
-        """ A tuple containing all parsed Forwarded header(s).
+        """A tuple containing all parsed Forwarded header(s).
 
         Makes an effort to parse Forwarded headers as specified by RFC 7239:
 
@@ -247,20 +258,13 @@ def scheme(self):
 
         'http' or 'https'.
         """
-        proto = None
+        scheme = self._scheme
+        if scheme is not None:
+            return scheme
         if self._transport.get_extra_info('sslcontext'):
-            proto = 'https'
-        elif self._secure_proxy_ssl_header is not None:
-            header, value = self._secure_proxy_ssl_header
-            if self.headers.get(header) == value:
-                proto = 'https'
+            return 'https'
         else:
-            proto = next(
-                (f['proto'] for f in self.forwarded if 'proto' in f), None
-            )
-            if not proto and hdrs.X_FORWARDED_PROTO in self._message.headers:
-                proto = self._message.headers[hdrs.X_FORWARDED_PROTO]
-        return proto or 'http'
+            return 'http'
 
     @property
     def method(self):
@@ -290,14 +294,14 @@ def host(self):
 
         Returns str, or None if no hostname is found in the headers.
         """
-        host = next(
-            (f['host'] for f in self.forwarded if 'host' in f), None
-        )
-        if host is None:
-            host = self._message.headers.get(hdrs.X_FORWARDED_HOST)
-        if host is None:
-            host = self._message.headers.get(hdrs.HOST)
-        return host
+        host = self._host
+        if host is not None:
+            return host
+        host = self._message.headers.get(hdrs.HOST)
+        if host is not None:
+            return host
+        else:
+            return socket.getfqdn()
 
     @reify
     def remote(self):
@@ -309,21 +313,15 @@ def remote(self):
         - X-Forwarded-For
         - peername of opened socket
         """
-        ip = next(
-            (f['for'] for f in self.forwarded if 'for' in f), None
-        )
-        if ip is None:
-            ips = self._message.headers.get(hdrs.X_FORWARDED_FOR)
-            if ips is not None:
-                ip = ips.split(',')[0].strip()
-        if ip is None:
-            transport = self._transport
-            peername = transport.get_extra_info('peername')
-            if isinstance(peername, (list, tuple)):
-                ip = peername[0]
-            else:
-                ip = peername
-        return ip
+        remote = self._remote
+        if remote is not None:
+            return remote
+        transport = self._transport
+        peername = transport.get_extra_info('peername')
+        if isinstance(peername, (list, tuple)):
+            return peername[0]
+        else:
+            return peername
 
     @reify
     def url(self):


Test patch:
diff --git a/aiohttp/test_utils.py b/aiohttp/test_utils.py
--- a/aiohttp/test_utils.py
+++ b/aiohttp/test_utils.py
@@ -502,7 +502,6 @@ def make_mocked_request(method, path, headers=None, *,
                         transport=sentinel,
                         payload=sentinel,
                         sslcontext=None,
-                        secure_proxy_ssl_header=None,
                         client_max_size=1024**2,
                         loop=...):
     """Creates mocked web.Request testing purposes.
@@ -570,7 +569,6 @@ def timeout(*args, **kw):
 
     req = Request(message, payload,
                   protocol, payload_writer, time_service, task, loop,
-                  secure_proxy_ssl_header=secure_proxy_ssl_header,
                   client_max_size=client_max_size)
 
     match_info = UrlMappingMatchInfo({}, mock.Mock())
diff --git a/docs/testing.rst b/docs/testing.rst
--- a/docs/testing.rst
+++ b/docs/testing.rst
@@ -374,7 +374,6 @@ conditions that hard to reproduce on real server::
                                   transport=sentinel, \
                                   payload=sentinel, \
                                   sslcontext=None, \
-                                  secure_proxy_ssl_header=None,
                                   loop=...)
 
    Creates mocked web.Request testing purposes.
@@ -414,12 +413,8 @@ conditions that hard to reproduce on real server::
    :param sslcontext: ssl.SSLContext object, for HTTPS connection
    :type sslcontext: ssl.SSLContext
 
-   :param secure_proxy_ssl_header: A tuple representing a HTTP header/value
-       combination that signifies a request is secure.
-   :type secure_proxy_ssl_header: tuple
-
    :param loop: An event loop instance, mocked loop by default.
-   :type secure_proxy_ssl_header: :class:`asyncio.AbstractEventLoop`
+   :type loop: :class:`asyncio.AbstractEventLoop`
 
    :return: :class:`aiohttp.web.Request` object.
 
diff --git a/tests/test_web_application.py b/tests/test_web_application.py
--- a/tests/test_web_application.py
+++ b/tests/test_web_application.py
@@ -203,27 +203,6 @@ def test_app_freeze():
     assert len(subapp.freeze.call_args_list) == 1
 
 
-def test_secure_proxy_ssl_header_default():
-    app = web.Application()
-    assert app._secure_proxy_ssl_header is None
-
-
-def test_secure_proxy_ssl_header_non_default(loop):
-    app = web.Application()
-    hdr = ('X-Forwarded-Proto', 'https')
-    app.make_handler(secure_proxy_ssl_header=hdr, loop=loop)
-    assert app._secure_proxy_ssl_header is hdr
-
-
-def test_secure_proxy_ssl_header_init(loop):
-    hdr = ('X-Forwarded-Proto', 'https')
-    with pytest.warns(DeprecationWarning):
-        app = web.Application(secure_proxy_ssl_header=hdr)
-    assert app._secure_proxy_ssl_header is hdr
-    app.make_handler(loop=loop)
-    assert app._secure_proxy_ssl_header is hdr
-
-
 def test_equality():
     app1 = web.Application()
     app2 = web.Application()
diff --git a/tests/test_web_request.py b/tests/test_web_request.py
--- a/tests/test_web_request.py
+++ b/tests/test_web_request.py
@@ -1,4 +1,5 @@
 import asyncio
+import socket
 from collections import MutableMapping
 from unittest import mock
 
@@ -17,7 +18,7 @@ def test_ctor():
 
     assert 'GET' == req.method
     assert HttpVersion(1, 1) == req.version
-    assert req.host is None
+    assert req.host == socket.getfqdn()
     assert '/path/to?a=1&b=2' == req.path_qs
     assert '/path/to' == req.path
     assert 'a=1&b=2' == req.query_string
@@ -229,24 +230,6 @@ def test_https_scheme_by_ssl_transport():
     assert req.secure is True
 
 
-def test_https_scheme_by_secure_proxy_ssl_header():
-    req = make_mocked_request('GET', '/',
-                              secure_proxy_ssl_header=('X-HEADER', '1'),
-                              headers={'X-HEADER': '1',
-                                       'Host': 'example.com'})
-    assert "https" == req.scheme
-    assert req.secure is True
-
-
-def test_https_scheme_by_secure_proxy_ssl_header_false_test():
-    req = make_mocked_request('GET', '/',
-                              secure_proxy_ssl_header=('X-HEADER', '1'),
-                              headers={'X-HEADER': '0',
-                                       'Host': 'example.com'})
-    assert "http" == req.scheme
-    assert req.secure is False
-
-
 def test_single_forwarded_header():
     header = 'by=identifier;for=identifier;host=identifier;proto=identifier'
     req = make_mocked_request('GET', '/',
@@ -388,61 +371,6 @@ def test_multiple_forwarded_headers_injection():
     assert req.forwarded[1]['by'] == '_actual_proxy'
 
 
-def test_https_scheme_by_forwarded_header():
-    header = 'by=_1;for=_2;host=_3;proto=https'
-    req = make_mocked_request('GET', '/',
-                              headers=CIMultiDict({'Forwarded': header}))
-    assert "https" == req.scheme
-    assert req.secure is True
-
-
-def test_https_scheme_by_malformed_forwarded_header():
-    req = make_mocked_request('GET', '/',
-                              headers=CIMultiDict({'Forwarded':
-                                                   'malformed value',
-                                                   'Host': 'xample.com'}))
-    assert "http" == req.scheme
-    assert req.secure is False
-
-
-def test_https_scheme_by_x_forwarded_proto_header():
-    req = make_mocked_request('GET', '/',
-                              headers={'X-Forwarded-Proto': 'https',
-                                       'Host': 'example.com'})
-    assert "https" == req.scheme
-    assert req.secure is True
-
-
-def test_https_scheme_by_x_forwarded_proto_header_no_tls():
-    req = make_mocked_request('GET', '/',
-                              headers={'X-Forwarded-Proto': 'http',
-                                       'Host': 'example.com'})
-    assert "http" == req.scheme
-    assert req.secure is False
-
-
-def test_host_by_forwarded_header():
-    headers = CIMultiDict()
-    headers.add('Forwarded', 'By=identifier1;for=identifier2, BY=identifier3')
-    headers.add('Forwarded', 'by=unknown;for=unknown;host=example.com')
-    req = make_mocked_request('GET', '/', headers=headers)
-    assert req.host == 'example.com'
-
-
-def test_host_by_forwarded_header_malformed():
-    req = make_mocked_request('GET', '/',
-                              headers=CIMultiDict({'Forwarded':
-                                                   'malformed value'}))
-    assert req.host is None
-
-
-def test_host_by_x_forwarded_host_header():
-    req = make_mocked_request('GET', '/',
-                              headers=CIMultiDict(
-                                  {'X-Forwarded-Host': 'example.com'}))
-    assert req.host == 'example.com'
-
-
 def test_host_by_host_header():
     req = make_mocked_request('GET', '/',
                               headers=CIMultiDict({'Host': 'example.com'}))
@@ -594,34 +522,6 @@ def test_remote_peername_unix():
     assert req.remote == '/path/to/sock'
 
 
-def test_remote_peername_x_forwarded():
-    transp = mock.Mock()
-    transp.get_extra_info.return_value = ('10.10.10.10', 1234)
-    req = make_mocked_request(
-        'GET', '/', transport=transp,
-        headers={'X-Forwarded-For': '11.11.11.11, 12.12.12.12'})
-    assert req.remote == '11.11.11.11'
-
-
-def test_remote_peername_forwarded():
-    transp = mock.Mock()
-    transp.get_extra_info.return_value = ('10.10.10.10', 1234)
-    req = make_mocked_request(
-        'GET', '/', transport=transp,
-        headers={'Forwarded': 'for=11.11.11.11, for=12.12.12.12'})
-    assert req.remote == '11.11.11.11'
-
-
-def test_remote_peername_forwarded_overrides_x_forwarded():
-    transp = mock.Mock()
-    transp.get_extra_info.return_value = ('10.10.10.10', 1234)
-    req = make_mocked_request(
-        'GET', '/', transport=transp,
-        headers={'Forwarded': 'for=11.11.11.11, for=12.12.12.12',
-                 'X-Forwarded-For': '13.13.13.13'})
-    assert req.remote == '11.11.11.11'
-
-
 def test_save_state_on_clone():
     req = make_mocked_request('GET', '/')
     req['key'] = 'val'
@@ -629,3 +529,21 @@ def test_save_state_on_clone():
     req2['key'] = 'val2'
     assert req['key'] == 'val'
     assert req2['key'] == 'val2'
+
+
+def test_clone_scheme():
+    req = make_mocked_request('GET', '/')
+    req2 = req.clone(scheme='https')
+    assert req2.scheme == 'https'
+
+
+def test_clone_host():
+    req = make_mocked_request('GET', '/')
+    req2 = req.clone(host='example.com')
+    assert req2.host == 'example.com'
+
+
+def test_clone_remote():
+    req = make_mocked_request('GET', '/')
+    req2 = req.clone(remote='11.11.11.11')
+    assert req2.remote == '11.11.11.11'