Repository: benoitc/gunicorn

Issue:
HTTPS Headers have "wsgi.url_scheme" set to "http" when using Unix socket between proxy and Gunicorn
Note: This issue follow the discussion from closed #1766 

Since version 19.8, when Gunicorn is started with option `--bind unix:/tmp/a_socket.sock`, request's headers are received with `wsgi.url_scheme` set to `http` instead of `https`.
As a result, some applications may wrongly consider the request is made an unsecure way.
This is the case with Django (for example) which have the `request.is_secure()` helper returning `False` when the website is opened from `https` URL.

There is a quick and dirty demo of this issue hosted on http://gunicorn-test.exige.info (http version) and https://gunicorn-test.exige.info (https version). Note : on some browsers, visiting https version first will block the ability to visit http version later.

The source code of this application is [available on GitHub](https://github.com/alorence/gunicorn-test). On this app, you can see the content of `request.META` and the content of all files used to run it (nginx and gunicorn config files, run.sh used to start gunicorn).
The important thing to note here is the value of `wsgi.url_scheme` in `request.environ` or in `request.META`. This variable has value `http` when visited from https URL.

While this issue was apparently resolved for some users (see [this comment from @zt-initech](https://github.com/benoitc/gunicorn/issues/1766#issuecomment-385489833)), there is many people still seeing this behavior (@predatell, @g10f, @benzkji, etc).
@tisdall I am ready to give you more information if you need.

Thanks a lot



Discussion:
I still cannot reproduce this.

First, I clone the repository and start the server using `DEVENV=true` so that it allows any host, as I will connect directly over the UNIX socket will cURL to test the behavior Gunicorn itself:

``` shellsession
> git clone https://github.com/alorence/gunicorn-test.git
Cloning into 'gunicorn-test'...
remote: Counting objects: 74, done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 74 (delta 22), reused 54 (delta 2), pack-reused 0
Unpacking objects: 100% (74/74), done.
> cd gunicorn-test
> pipenv install -r requirements.txt
Creating a virtualenv for this project...
Pipfile: /Users/randall/Desktop/gunicorn-test/Pipfile
Using /usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7 (3.7.0) to create virtualenv...
⠋Already using interpreter /usr/local/Cellar/pipenv/2018.7.1/libexec/bin/python3.7
Using real prefix '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/randall/.local/share/virtualenvs/gunicorn-test-yjWQO51Y/bin/python3.7
Also creating executable in /Users/randall/.local/share/virtualenvs/gunicorn-test-yjWQO51Y/bin/python
Installing setuptools, pip, wheel...done.
Setting project for gunicorn-test-yjWQO51Y to /Users/randall/Desktop/gunicorn-test

Virtualenv location: /Users/randall/.local/share/virtualenvs/gunicorn-test-yjWQO51Y
Creating a Pipfile for this project...
Requirements file provided! Importing into Pipfile...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (847131)!
Installing dependencies from Pipfile.lock (847131)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/4 — 00:00:07
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
> pipenv shell
Launching subshell in virtual environment…
 . /Users/randall/.local/share/virtualenvs/gunicorn-test-yjWQO51Y/bin/activate
(gunicorn-test-yjWQO51Y) > chmod +x run.sh
(gunicorn-test-yjWQO51Y) > DEVENV=true ./run.sh
./run.sh: line 3: venv/bin/activate: No such file or directory
Operations to perform:
  Apply all migrations: contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying sessions.0001_initial... OK
[2018-08-25 14:23:51 -0700] [64030] [INFO] Starting gunicorn 19.9.0
[2018-08-25 14:23:51 -0700] [64030] [INFO] Listening at: unix:/var/tmp/gunicorn_tests.sock (64030)
[2018-08-25 14:23:51 -0700] [64030] [INFO] Using worker: sync
[2018-08-25 14:23:51 -0700] [64033] [INFO] Booting worker with pid: 64033
```

Then, in another terminal, I make a request and check the output:

``` shellsession
> curl -s --unix-socket /var/tmp/gunicorn_tests.sock -H "X-Forwarded-Proto: https" http://localhost/ | grep 'request.*wsgi.*url_scheme'
                    request.environ['wsgi.url_scheme']: https<br/>
```

If you can reproduce the behavior I'm seeing, then the issue might be between nginx and gunicorn, or there is some difference in platform (I am on macOS).
Interestingly, on my Debian 9 server, with the exact same steps (`DEVENV=true`) I still have `wsgi.url_scheme` set to `http`

```
~ ᐅ curl -s --unix-socket /var/tmp/gunicorn_tests.sock -H "X-Forwarded-Proto: https" http://localhost/ | grep 'request.*wsgi.*url_scheme'
                    request.environ['wsgi.url_scheme']: http<br/>
```

I also tried leaving `DEVENV` with its default value `False`, and modifying `ALLOWED_HOSTS = ["localhost", "127.0.0.1"]` in `settings.py`. The result is the same:

```
~ ᐅ curl -s --unix-socket /var/tmp/gunicorn_tests.sock -H "X-Forwarded-Proto: https" http://localhost/ | grep 'request.*wsgi.*url_scheme'
                    request.environ['wsgi.url_scheme']: http<br/>
```

Very interesting. I'll check this out as soon as I can.
Alright, I can repro it in a Docker environment. I'll see what I can find.
The difference seems to be that `socket.getpeername()` returns a `bytes` on Python3 in my Debian container while it returns a `str` on macOS.

The [`socket` documentation](https://docs.python.org/3/library/socket.html#socket-families) states:

> The address of an AF_UNIX socket bound to a file system node is represented as a string, using the file system encoding and the 'surrogateescape' error handler (see PEP 383).

In the Debian case, this agrees with `sys.getfilesystemencoding()` returning `ascii` while in the macOS case it returns `utf-8`, so this could be the difference. What's strange is that if I set the system locale to `en_US.UTF-8`, `sys.getfilesystemencoding()` returns `utf-8` but the socket name is still a `bytes` instance.

The documentation does say that "programs intended to run on Linux may need to deal with both types of address".

I think the right course of action is probably to change everywhere we test for UNIX sockets to deal with either type, or better yet to follow the advice from [this comment](https://github.com/benoitc/gunicorn/issues/1766#issuecomment-385489833) that I should have heeded the first time.

Resolving patch:
diff --git a/gunicorn/http/message.py b/gunicorn/http/message.py
--- a/gunicorn/http/message.py
+++ b/gunicorn/http/message.py
@@ -71,11 +71,11 @@ def parse_headers(self, data):
             secure_scheme_headers = cfg.secure_scheme_headers
         elif isinstance(self.unreader, SocketUnreader):
             remote_addr = self.unreader.sock.getpeername()
-            if isinstance(remote_addr, tuple):
+            if self.unreader.sock.family in (socket.AF_INET, socket.AF_INET6):
                 remote_host = remote_addr[0]
                 if remote_host in cfg.forwarded_allow_ips:
                     secure_scheme_headers = cfg.secure_scheme_headers
-            elif isinstance(remote_addr, str):
+            elif self.unreader.sock.family == socket.AF_UNIX:
                 secure_scheme_headers = cfg.secure_scheme_headers
 
         # Parse headers into key/value pairs paying attention
diff --git a/gunicorn/sock.py b/gunicorn/sock.py
--- a/gunicorn/sock.py
+++ b/gunicorn/sock.py
@@ -132,7 +132,7 @@ def _sock_type(addr):
             sock_type = TCP6Socket
         else:
             sock_type = TCPSocket
-    elif isinstance(addr, str):
+    elif isinstance(addr, (str, bytes)):
         sock_type = UnixSocket
     else:
         raise TypeError("Unable to create socket from: %r" % addr)


Test patch:
diff --git a/tests/test_sock.py b/tests/test_sock.py
--- a/tests/test_sock.py
+++ b/tests/test_sock.py
@@ -11,6 +11,27 @@
 from gunicorn import sock
 
 
+@mock.patch('os.stat')
+def test_create_sockets_unix_bytes(stat):
+    conf = mock.Mock(address=[b'127.0.0.1:8000'])
+    log = mock.Mock()
+    with mock.patch.object(sock.UnixSocket, '__init__', lambda *args: None):
+        listeners = sock.create_sockets(conf, log)
+        assert len(listeners) == 1
+        print(type(listeners[0]))
+        assert isinstance(listeners[0], sock.UnixSocket)
+
+
+@mock.patch('os.stat')
+def test_create_sockets_unix_strings(stat):
+    conf = mock.Mock(address=['127.0.0.1:8000'])
+    log = mock.Mock()
+    with mock.patch.object(sock.UnixSocket, '__init__', lambda *args: None):
+        listeners = sock.create_sockets(conf, log)
+        assert len(listeners) == 1
+        assert isinstance(listeners[0], sock.UnixSocket)
+
+
 def test_socket_close():
     listener1 = mock.Mock()
     listener1.getsockname.return_value = ('127.0.0.1', '80')