Repository: mitmproxy/mitmproxy

Issue:
Inability to connect to ipv6 websites through mitmproxy from ipv6 network.
##### Steps to reproduce the problem:

1. Follow [instructions](https://github.com/mitmproxy/mitmproxy/blob/master/docs/transparent/linux.rst) to setup transparent proxy on Linux on my own machine, substituting PREROUTING in the iptables ruleset with OUTPUT.
2. Run `mitmdump --mode transparent`
3. Navigate to ipv4 site (e.g. espn.com) and ipv6 site (e.g. facebook.com).

I've done this from two networks. On one network, I have an ipv6 address (according to http://test-ipv6.com/). On the other network, I do not have an ipv6 address.

+ ipv4 network: Able to connect to both espn.com and facebook.com
+ ipv6 network: Only able to connect to espn.com. Additionally, when I use http://test-ipv6.com after doing the first two steps above, the website can no longer find my ipv6 address.

On the ipv6 network, I receive a message saying the following:

```
Client connection was killed because allow_remote option is set to false, client IP was not a private IP and proxyauth was not set.
To allow remote connections set allow_remote option to true or set proxyauth option.
[2601:246:100:cefc:7da3:3d6c:6195:5f1f]:42868: Connection killed
[2601:246:100:cefc:7da3:3d6c:6195:5f1f]:42868: clientdisconnect
```

When I set the allow_remote option to be true (`mitmproxy --mode transparent --set allow_remote=true`), I now receive the following message:

```
[2601:246:100:cefc:7da3:3d6c:6195:5f1f]:57032: Transparent mode failure: FileNotFoundError(2, ‘No such file or directory’)
```

##### Any other comments? What have you tried so far?

I should try putting mitmproxy in a VM or Docker container. Then, I think I would instead have to use the PREROUTING chain. If the transparent proxy works from the VM/Docker container, this would indicate that the problem has to do with configuration when using the OUTPUT chain.

I think it's reasonable that mitmproxy should support ipv6 when run on your own computer. If we can figure out what the problem is, I'd be happy to submit a pull request to the Linux transparent proxy documentation with more explicit instructions on how to set up mitmproxy. Apologies if this is a better fit in discourse, but I think this issue at the very least reflects something which could be improved in the transparent proxy documentation.

One hacky option would be to shut down ipv6 altogether through configuration, but I haven't figured out how to make it work. Another possibility is that this has to do with [router advertisements](http://strugglers.net/~andy/blog/2011/09/04/linux-ipv6-router-advertisements-and-forwarding/), but simply setting the accept_ra flag to be equal to two did not work for me. 

##### System information

Mitmproxy: 3.0.0.dev1179 (commit 6d8731e) 
Python:    3.6.4
OpenSSL:   OpenSSL 1.1.0g  2 Nov 2017
Platform:  Linux-4.4.0-21-generic-x86_64-with-debian-stretch-sid


<!-- Please use the mitmproxy forums (https://discourse.mitmproxy.org/) for support/how-to questions. Thanks! :) -->



Discussion:
Thanks for the fantastic write-up! The `FileNotFoundError` in transparent mode is a really weird one, because the only function that could trigger that is `mitmproxy.platform.linux.original_addr`, which doesn't open any files... Can you verify that this is indeed triggered by [linux.py#L9](https://github.com/mitmproxy/mitmproxy/blob/baf4b5dc03c3b87fbef6e5df422eb7654594978f/mitmproxy/platform/linux.py#L9)?


I think I made an important omission in writing up the issue, which is that I've usually been passing a script to mitmproxy, and this write up is based off errors received when passing the script. So, it's more likely that this is being triggered [here](https://github.com/mitmproxy/mitmproxy/blob/6d8731e144eaec2aaf2120c21969c9228b7af95a/mitmproxy/addons/script.py). It's still not obvious how ipv4/ipv6 would affect triggering the error here.

I'm pretty sure I also encountered problems when not passing the script, but I'm now I'm a bit less confident. I'll have access to the ipv6 network tonight, and I'll try it again with and without a script.
Not sure what your script does, but the code path for "Transparent mode failure" is pretty clear:

https://github.com/mitmproxy/mitmproxy/blob/568f40c810f4de60f10bd814608fde8268ef7733/mitmproxy/proxy/modes/transparent_proxy.py#L12-L15
https://github.com/mitmproxy/mitmproxy/blob/568f40c810f4de60f10bd814608fde8268ef7733/mitmproxy/platform/__init__.py#L20-L23
https://github.com/mitmproxy/mitmproxy/blob/568f40c810f4de60f10bd814608fde8268ef7733/mitmproxy/platform/linux.py#L8-L12
You're right - I got fixated on the FileNotFoundError, since the only place it shows up in the code is in [script.py](https://github.com/mitmproxy/mitmproxy/blob/6d8731e144eaec2aaf2120c21969c9228b7af95a/mitmproxy/addons/script.py).

[The script](https://github.com/aniketpanjwani/chomper/blob/master/chomper/filter.py) just adjusts the response for particular URLS.
So, I'm not entirely sure how to verify that it is being triggered by linux.py. Is there some sort of debug mode I can use to drop down to pdb in case of an error?

I have two additional things to report:

1. I followed [these instructions](https://community.rackspace.com/products/f/public-cloud-forum/5110/how-to-prefer-ipv4-over-ipv6-in-ubuntu-and-centos) to get my operating system to prefer ipv4 over ipv6. After following these instructions, I am now able to see in mitmproxy all the dual-stack websites such as facebook and instagram which I was previously not seeing. 
2. [One VPN](https://helpdesk.privateinternetaccess.com/hc/en-us/articles/232324908-Why-Do-You-Block-IPv6-) claims that they "do not use ipv6 as the current implementations are insecure and could potentially compromise your privacy and security", and that "most VPN software fails to direct IPv6 traffic through the VPN tunnel, so when you connect to an IPv6 enabled website, your browser will make an IPv6 DNS request outside the VPN, which is therefore handled by your ISP". They don't really explain why this occurs, but it's possible that whatever is inherent in ipv6 that makes its use difficult with VPNs is also the issue preventing its use as a transparent proxy. I really know very little about computer networks, so I'm just reasoning by induction.
> So, I'm not entirely sure how to verify that it is being triggered by linux.py. Is there some sort of debug mode I can use to drop down to pdb in case of an error?

Assuming you have installed mitmproxy from source, the easiest form of debugging would be to add print statements in linux.py. pdb works as well of course, you can call that from an addon in mitmdump so that the UI is out of the way or use https://github.com/mitmproxy/mitmproxy/blob/master/examples/complex/remote_debug.py.

And no, there's no particular reason why IPv6 DNS should behave differently. It's very likely possible that our `getsockopt` call doesn't work for IPv6, but I'm quite confused why that triggers a FileNotFound.
Ah, it looks like @chadbrubaker was ahead of us: https://github.com/google/nogotofail/commit/c73070ca3098faab2994a83a9249d7c973151aa4. 
Could you check if applying this fix solves it?
Oh cool! I'll get to this in the next few hours.
Getting this error when installing from source as per [instructions](https://github.com/mitmproxy/mitmproxy#development-setup):

```
+ echo Creating dev environment in ./venv...
Creating dev environment in ./venv...
+ python3 -m venv venv
 Error: Command '['/home/aniket/personal/other/mitmproxy/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.
```
If I install the requirements from pipenv, will it still be the case that mitmproxy is installed as "editable" (so that it responds to my pdb entries)? I'll try it out...
I got the environment working. Here's what I tried - copying @chadbrubaker as closely as possible:

```
import socket
import struct

# Python socket module does not have this constant
SO_ORIGINAL_DST = 80


def original_addr(csock: socket.socket):
    family = csock.family
    if family == socket.AF_IFNET:
        odestdata = csock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 28)
    elif family == socket.AF_INET6:
        SOL_IPV6 = 41
        odestdata = csock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 64)
    _, port, a1, a2, a3, a4 = struct.unpack("!HHBBBBxxxxxxxx", odestdata)
    address = "%d.%d.%d.%d" % (a1, a2, a3, a4)
    return address, port
```
I get an error saying `'socket' has no attribute 'AF_INET'`. Any thoughts?

Also, when I stick a `pdb.set_trace()` inside linux.py, I'm able to find `socket.AF_INET`. Some sort of namespace error going on...
`if family == socket.AF_IFNET:` <- typo
Fixed the typo, but still getting the same TransparentModeFailure.
Still a FileNotFound?
 Yep.
Can you get on Slack so that we can shorten turnaround times?
Sure thing.
Ok, this turned out to be much more of a rabbit hole than I expected. 🐇

1. `sock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28)` is generally the right thing we need to do to support IPv6.
2. Surprise: On IPv6 networks this now breaks IPv4 connections: 
 `sock.family` is `AF_INET6`, `peername` is an IPv4-mapped `::ffff:192.168.0.102`, and the syscall fails. 
If we observe an IPv4-mapped IPv6 address, we do need to use the original IPv4 syscall. 🙃
3. Biggest surprise: If we call `getsockopt` with arguments for the wrong address type, @aniketpanjwani can reliably crash his Python runtime. Example code:
```python
    try:
        dst = sock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
        print("IPv4: {}".format(dst))
        port, raw_ip = struct.unpack_from("!2xH4s", dst)
        ip = socket.inet_ntop(socket.AF_INET, raw_ip)
    except Exception:
        dst = sock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28)
        print("IPv4 failed, IPv6: {}".format(dst))
        port, raw_ip = struct.unpack_from("!2xH4x16s", dst)
        ip = socket.inet_ntop(socket.AF_INET6, raw_ip)
```
4. Overall, the solution seems to be something like this:
```python
import socket
import struct
import typing

# Python's socket module does not have these constants
SO_ORIGINAL_DST = 80
SOL_IPV6 = 41


def original_addr(sock: socket.socket) -> typing.Tuple[str, int]:
    # Even if sock.family is AF_INET6, we can't be sure that we actually
    # need to call getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28):
    # For some reason we may have to deal with IPv4-mapped IPv6 addresses here,
    # in which case only the IPv4 getsockopt call works.
    # As an additional bonus, we can't just try the IPv4 one and then do IPv6 if that doesn't work,
    # because doing the wrong call apparently crashes the whole Python runtime.
    # As such, we use getsockname() as a heuristic to check if we need to go with IPv4.
    is_ipv4 = "." in sock.getsockname()[0]  # either 127.0.0.1 or ::ffff:127.0.0.1
    if is_ipv4:
        # the struct returned here should only have 8 bytes, but invoking with buflen=8 doesn't work.
        dst = sock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
        port, raw_ip = struct.unpack_from("!2xH4s", dst)
        ip = socket.inet_ntop(socket.AF_INET, raw_ip)
    else:
        dst = sock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28)
        port, raw_ip = struct.unpack_from("!2xH4x16s", dst)
        ip = socket.inet_ntop(socket.AF_INET6, raw_ip)
    return ip, port
```

I'll check this tomorrow with a fresh mind and then add tests and submit a PR. Thanks again for the help debugging, @aniketpanjwani! 🍰 

Resolving patch:
diff --git a/mitmproxy/platform/linux.py b/mitmproxy/platform/linux.py
--- a/mitmproxy/platform/linux.py
+++ b/mitmproxy/platform/linux.py
@@ -1,12 +1,34 @@
 import socket
 import struct
+import typing
 
-# Python socket module does not have this constant
+# Python's socket module does not have these constants
 SO_ORIGINAL_DST = 80
+SOL_IPV6 = 41
 
 
-def original_addr(csock: socket.socket):
-    odestdata = csock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
-    _, port, a1, a2, a3, a4 = struct.unpack("!HHBBBBxxxxxxxx", odestdata)
-    address = "%d.%d.%d.%d" % (a1, a2, a3, a4)
-    return address, port
+def original_addr(csock: socket.socket) -> typing.Tuple[str, int]:
+    # Get the original destination on Linux.
+    # In theory, this can be done using the following syscalls:
+    #     sock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
+    #     sock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28)
+    #
+    # In practice, it is a bit more complex:
+    #  1. We cannot rely on sock.family to decide which syscall to use because of IPv4-mapped
+    #     IPv6 addresses. If sock.family is AF_INET6 while sock.getsockname() is ::ffff:127.0.0.1,
+    #     we need to call the IPv4 version to get a result.
+    #  2. We can't just try the IPv4 syscall and then do IPv6 if that doesn't work,
+    #     because doing the wrong syscall can apparently crash the whole Python runtime.
+    # As such, we use a heuristic to check which syscall to do.
+    is_ipv4 = "." in csock.getsockname()[0]  # either 127.0.0.1 or ::ffff:127.0.0.1
+    if is_ipv4:
+        # the struct returned here should only have 8 bytes, but invoking sock.getsockopt
+        # with buflen=8 doesn't work.
+        dst = csock.getsockopt(socket.SOL_IP, SO_ORIGINAL_DST, 16)
+        port, raw_ip = struct.unpack_from("!2xH4s", dst)
+        ip = socket.inet_ntop(socket.AF_INET, raw_ip)
+    else:
+        dst = csock.getsockopt(SOL_IPV6, SO_ORIGINAL_DST, 28)
+        port, raw_ip = struct.unpack_from("!2xH4x16s", dst)
+        ip = socket.inet_ntop(socket.AF_INET6, raw_ip)
+    return ip, port


Test patch: