requests doen't work with IPv6 addresses that contains the '%' sign

Some IPv6 addresses contains the '%' sign ( link-local addresses : ex. fe80::21b:63ff:feab:e6a6%eth0 or [fe80::21b:63ff:feab:e6a6%eth0] )
When using this addresses the requests library raises the fallowing exception: "requests.exceptions.InvalidURL: Invalid percent-escape sequence: 'et'"
(raised by the function unquote_unreserved() from the utils.py).
If I put as first line in this function the "return uri" line (in other words if I pass over this function), everything works ok.


The % character is reserved in a URI. This causes us no end of trouble because people expect us to magically know whether we should leave a % sequence alone or transform the % to its percent-encoded form. Realistically, we have no way of being able to do that correctly.

Our concession is to say that we can sort out everything _except_ encoding percent characters: those need to be encoded by the user themselves. You'll want to replace the % in the IPv6 URI with its percent-encoded form (%25).


Hmm.. yes now I see that the '%' character is reserved in a URI.. Anyway, I tried replacing it with %25 (http://[fe80::240:8cff:fec5:845e%25eth0]) and it still dosn't work, because the '%' characted is not in UNRESERVED_SET, '%25' is not transformed into '%', it is kept as it is.


That's becuase, as discussed, it's reserved and therefore should not be in the UNRESERVED_SET. In what way does your request fail?


It fails with the fallowing error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='fe80::240:8cff:fec5:845e%25eth0', port=80): Max retries exceeded with url: /axis-cgi/param.cgi?action=list&group=root.AutoTracking.A0.Running (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)

That is because the unquote_unreserved() function doesn't change anything in the URL, it should have replaced '%25' with '%' but it doesn't, it just returns the URL as it is.


It should _not_ have replaced '%25' with '%'. You really need to read up on the rules for constructing URLs, but allow me to reproduce them for you here:

The host portion of the URL is defined as follows:

```
host        = IP-literal / IPv4address / reg-name

IP-literal  = "[" ( IPv6address / IPvFuture  ) "]"
IPvFuture   = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )

IPv6address =                            6( h16 ":" ) ls32
            /                       "::" 5( h16 ":" ) ls32
            / [               h16 ] "::" 4( h16 ":" ) ls32
            / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
            / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
            / [ *3( h16 ":" ) h16 ] "::"    h16 ":"   ls32
            / [ *4( h16 ":" ) h16 ] "::"              ls32
            / [ *5( h16 ":" ) h16 ] "::"              h16
            / [ *6( h16 ":" ) h16 ] "::"

ls32        = ( h16 ":" h16 ) / IPv4address
            ; least-significant 32 bits of address

h16         = 1*4HEXDIG
            ; 16 bits of address represented in hexadecimal

reg-name    = *( unreserved / pct-encoded / sub-delims )
```

Let's go through these in turn. Is the address you've given (`[fe80::240:8cff:fec5:845e%25eth0]`) a valid IPv6 address, as defined above? No, it is not, because it contains a percent character which is not allowed by the above rules.

Is it an IPvFuture? No, it is not, because it contains a percent character, which is not in `unreserved`, `sub-delims` or `HEXDIGIT`.

Is it a reg-name? No, it is not, because `pct-encoded` is defined as `pct-encoded = "%" HEXDIG HEXDIG` and `t` is not a HEXDIGIT.

The only way it's a valid URL is if the percent character is percent encoded. This is why we didn't unencode it: we'd end up with an invalid URL.

Let's ask a different question: does this work in the standard library?


Yes, it does work with httplib, for example:
conn = httplib.HTTPConnection("[fe80::240:8cff:fec5:845e%eth0]")


Interesting. Does it work in urllib3?


So the key problem here is that the URL syntax above was supplemented in RFC 6874 to allow the percent character. I suspect our URL parsing logic is invalid here.


comment: In some environments IPv6 Link Local addressing is VERY useful. If you would fix it, that would be helpful.


@steve100 can you test if the `rfc3986` library handles this appropriately?


what do I have to load to test the rfc3986 library?  I probably can test it if you give me some more details.  Thanks.


I am also stuck on this issue. I can also pitch in if needed.


if someone tells me how to test with this library:
I can.  rfc3986

On Fri, Mar 11, 2016 at 1:42 AM, littypreeth notifications@github.com
wrote:

> I am also stuck on this issue. I can also pitch in if needed.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/1985#issuecomment-195219184
> .


I had forgotten that there's already an open issue for this feature within rfc3986: https://github.com/sigmavirus24/rfc3986/issues/2 so there's no need to test it.


Thanks.

I know the issues that surround using the IPv6 Link Local Address..

Here it is   SO tempting to use the link local address help get a device or
vm configured with better addresses and other configuration information.

Steve

On Fri, Mar 11, 2016 at 2:32 PM, Ian Cordasco notifications@github.com
wrote:

> I had forgotten that there's already an open issue for this feature within
> rfc3986: sigmavirus24/rfc3986#2
> https://github.com/sigmavirus24/rfc3986/issues/2 so there's no need to
> test it.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/1985#issuecomment-195512868
> .


Old thread, but I'm running into this LL IPv6 % handling issue and wondering if was ever resolved?  people are (successfully) using curl instead of requests and it breaks my heart.  I'm using python 2.7.11 and requests 2.9.1


No. It's not resolved yet.

On 17-Aug-2016 9:47 pm, "briggr1" notifications@github.com wrote:

> Old thread, but I'm running into this LL IPv6 % handling issue and
> wondering if was ever resolved? people are (successfully) using curl
> instead of requests and it breaks my heart. I'm using python 2.7.11 and
> requests 2.9.1
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/1985#issuecomment-240464306,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AQ9F5Qht1dy6A2VFw1UN_htNSCgf_dnOks5qgzQFgaJpZM4BuSvZ
> .


updated to python 2.7.12, and requests 2.11.0 - no dice.


So I'm going to start working on adding this support to [rfc3986](/sigmavirus24/rfc3986). In reading the specification, there's this:

>   In a URI, a literal IPv6 address is always embedded between "[" and
>   "]".  This document specifies how a <zone_id> can be appended to the
>   address.  According to URI syntax [RFC3986], "%" is always treated as
>   an escape character in a URI, so, according to the established URI
>   syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
>   be percent-encoded and represented in the form "%25".  Thus, the
>   scoped address fe80::a%en1 would appear in a URI as
>   http://[fe80::a%25en1].

So the right thing to do is encode the `%` sign it seems.

@Lukasa should we reopen this to track the work that we need to do in urllib3?


In the first instance let's just open a relevant issue on urllib3.


Hi @Lukasa

Wondering if you could post the related issue number for this on urllib3 so I could track it?  I tried looking for it, but didn't see it (I did see something about proxy handling...).  Thanks in advance

Also, for others that might be wondering - there is a workaround: if your system only has 1 interface with IPv6 addressing, then you do not need to supply the %<interface>, and things work just fine.  Unfortunately, all of my testing involves many networks with different vlans and I need IPv6 support on them all.


@briggr1 I don't believe any tracking issue for this was actually opened on urllib3. I'd need @sigmavirus24 to outline exactly what work he believes is needed though: in my initial testing urllib3 seems to be handling this appropriately.