Error 404 for url, that contains relative path parts

Browser & other tools seem to ignore trailing dot in the URL. Unfortunately `requests` does not.

Compare

```
$ curl -s https://github.com/. -o /dev/null -w "%{http_code}"; echo
200
```

With 

```
$ python -c 'import requests; print(requests.get("https://github.com/.").status_code)'
404
```


requests/api.py - request()

```
with sessions.Session() as session:
    return session.request(method=method, url=url.rstrip("."), **kwargs)
```

Not sure if this matches any style guides, but should help solve this issue


@TetraEtc that's overly general. While curl and browsers detect the `.` as the sole element of the path, they don't do anything fancy for `https://github.com/foo.` which yields a 404.

Further Requests is an HTTP client, not an oracle or a browser. We can't know what URLs to meaningfully trim/munge/etc. unless a specification tells us to do so. Even then I'd air on the side of not doing this behind the scenes for the user.


You are not exactly right.

https://github.com/kennethreitz/requests/. works.
Even  https://github.com/kennethreitz/blahblah/../requests/ works.

That is called relative URIs

See [some RFC](http://tools.ietf.org/html/rfc2396#section-5)


@Lol4t0 
From the RFC you referenced (which is btw deprecated, the new one says the [same](http://tools.ietf.org/html/rfc3986#section-3.3)):

```
  Although this is
   very similar to their use within Unix-based filesystems to indicate
   directory levels, these path components are only considered special
   when resolving a relative-path reference to its absolute form
```

`only considered special when resolving a relative-path...`. requests does not resolve a path, it uses is as is.


It have to resolve at least to perform redirects
28 янв. 2016 г. 2:51 пользователь "Thomas Weißschuh" <
notifications@github.com> написал:

> @Lol4t0 https://github.com/Lol4t0
> From the RFC you referenced (which is btw deprecated, the new one says the
> same http://tools.ietf.org/html/rfc3986#section-3.3):
> 
>   Although this is
>    very similar to their use within Unix-based filesystems to indicate
>    directory levels, these path components are only considered special
>    when resolving a relative-path reference to its absolute form
> 
> only considered special when resolving a relative-path.... requests does
> not resolve a path, it uses is as is.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2982#issuecomment-175914604
> .


@Lol4t0 as @t-8ch points out, relative-references are not what requests accepts (we only accept absolute references which is what you passed us).

> It have to resolve at least to perform redirects

What?


@Lol4t0 those links work in your browser (reminder, requests is not a browser)

```
❯❯❯ curl -i -vv -s 'https://github.com/kennethreitz/requests.'
*   Trying 192.30.252.128...
* Connected to github.com (192.30.252.128) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: github.com
* Server certificate: DigiCert SHA2 Extended Validation Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET /kennethreitz/requests. HTTP/1.1
> Host: github.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
HTTP/1.1 404 Not Found
< Server: GitHub.com
Server: GitHub.com
< Date: Wed, 27 Jan 2016 23:55:32 GMT
Date: Wed, 27 Jan 2016 23:55:32 GMT
< Content-Type: application/json; charset=utf-8
Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Status: 404 Not Found
Status: 404 Not Found
< Cache-Control: no-cache
Cache-Control: no-cache
< Vary: X-PJAX
Vary: X-PJAX
< X-UA-Compatible: IE=Edge,chrome=1
X-UA-Compatible: IE=Edge,chrome=1
< X-Request-Id: 312fc05db0965f607e692a21c1217b43
X-Request-Id: 312fc05db0965f607e692a21c1217b43
< X-Runtime: 0.007642
X-Runtime: 0.007642
< Content-Security-Policy: default-src 'none'; base-uri 'self'; connect-src 'self'; form-action 'self'; img-src data:; script-src 'self'; style-src 'unsafe-inline'
Content-Security-Policy: default-src 'none'; base-uri 'self'; connect-src 'self'; form-action 'self'; img-src data:; script-src 'self'; style-src 'unsafe-inline'
< Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
< Public-Key-Pins: max-age=300; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="JbQbUG5JMJUoI6brnx0x3vZF6jilxsapbXGVfjhN8Fg="; includeSubDomains
Public-Key-Pins: max-age=300; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="JbQbUG5JMJUoI6brnx0x3vZF6jilxsapbXGVfjhN8Fg="; includeSubDomains
< X-Content-Type-Options: nosniff
X-Content-Type-Options: nosniff
< X-Frame-Options: deny
X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block
< X-GitHub-Request-Id: 48A0CA7F:4757:60D2E69:56A958F4
X-GitHub-Request-Id: 48A0CA7F:4757:60D2E69:56A958F4

<
* Connection #0 to host github.com left intact
{"error":"Not Found"}
```

They don't work with curl though.


Server can return you a relative URL as redirect target with 302 for
example. How are you going to handle that?
L
28 янв. 2016 г. 2:55 пользователь "Ian Cordasco" notifications@github.com
написал:

> @Lol4t0 https://github.com/Lol4t0 as @t-8ch https://github.com/t-8ch
> points out, relative-references are not what requests accepts (we only
> accept absolute references which is what you passed us).
> 
> It have to resolve at least to perform redirects
> 
> What?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2982#issuecomment-175915919
> .


Note even if we did start to use [rfc3986](/sigmavirus24/rfc3986) this wouldn't be handled because (as discussed above) these are not relative references. These are absolute references. 


@Lol4t0 relative Location headers are different. We already handle relative Location headers although I don't think we've tried handling any with a `.` or `..` in them. You're changing the bug report though. User-input will not be modified. We'll investigate our handling of relative Location headers with `.` or `..` in them.


1. May be you copy-paste correctly in curl and try again? (`requests. != requests/.`, first is not relative in any sense)
2. Those are valid URLs. User input should not be changed, but URL should be simplified before sending request. That is not any different from URL encoding

I believe requests should work consistently with other tools anyway. Note that curl is the same level to requests, and it simplifies urls. More, handling Location and direct input differently makes requests inconsistent with itself.


> More, handling Location and direct input differently makes requests inconsistent with itself.

I completely disagree. One (the Location header provided by the server) is an _actual_ relative reference. The one you're providing is an absolute reference. There's nothing inconsistent with resolving relative references provided by the server and not munging an absolute reference provided by the user.


> absolute reference provided by the user

If you look into 5.1, 5.2.2, you'll see, that to resolve an URL it should first be converted to absolute form, and that relative URL can contain scheme, host, etc. Strict rules of conversion also provided, so you didn't have to be oracle to make conversion.

If you tell me that you are not going to resolve URLs, I'm asking _who_ should do it? Doesn't it mean that _every_ URL that has some sort of foreign input (i.e. user input, configuration file, obtained as a result of another request, so, just _most_ of) should be resolved _additionally_ before using requests, that is sort of _super verbose and should not be in python_?


This is a remarkably tricky issue, but in general I'm sympathetic to the idea that requests should normalise the URL as much as possible when provided by the user or when received in the Location header.

I appreciate that generally speaking we try not to manipulate the URLs provided by users _too_ extensively, but we do still play with them: we try to urlencode partially encoded URLs and generally just try to make sense of what the remote party has sent us. For that reason, I also think we should take a similar attitude to partial paths.

However, there are some reasons to be tentative here. The first is that this amounts to a major change, which would mean it has to go in to 3.0.0. The second is that if we're determined to start handling URLs "properly", we should aim to handle them _really_ properly. I believe @glyph is working on a URL handling library, but generally speaking I think we should be aiming to bring in something that lets us work with URLs more effectively so that we can do something that amounts to appropriate behaviour.

The upshot means that, if @sigmavirus24 agrees with me, we'll need to put this on the backburner until we get a library that can let us work with URLs more effectively.


Could you please refer to the @glyph 's repository so I could use it meanwhile it is not integrated into requests or participate in its development may be


Currently I think @glyph is developing it as part of Twisted, so I'm honestly not sure where it lives. However, I'm sure @glyph would be delighted to weigh into this thread at some time today. =)


It's been part of a Twisted release, in fact, so you can `pip install twisted` and play with it.

You can see the documentation here:

https://twistedmatrix.com/documents/15.5.0/api/twisted.python.url.URL.html

This module is part of Twisted currently, but:
1. the plan is to spin it out into a separate library, and release it on PyPI (probably as `pip install iri`, since it is concerned largely with proper representation and transcoding of internationalized resource identifiers)
2. the module is already totally self-contained; since `requests` already vendors absolutely everything, feel free to vendor this code out of a Twisted source release; it will work fine.  It even goes so far as to avoid importing Twisted's own `TestCase` subclass, so that the unit tests will be portable.
3. I wanted to leave it in Twisted to let some people experiment with it before putting the API in stone, since I assume if we actually do a separate PyPI release it would firm up the interface.  However, thus far I've been very happy with the interface, and so if requests wanted to make use of it as-is, I'd be happy to harden its compatibility guarantees so you can pass them on to your users.


Thanks @glyph. I'll try to play around with this at some point and see if it's useful for requests.


> I appreciate that generally speaking we try not to manipulate the URLs provided by users too extensively, but we do still play with them: we try to urlencode partially encoded URLs and generally just try to make sense of what the remote party has sent us. For that reason, I also think we should take a similar attitude to partial paths.

The way that `URL` deals with this tension is as follows:

When you call `URL.fromText`, the parser attempts to retain as much information as possible from the text that you have passed to it.  `URL` intentionally does _not_ have a `fromBytes` method, because at a bare minimum, URLs always need to be representable as text that can be displayed to a human, because you have to put these things into URL bars or logs or somesuch, and those are textual data.

If you want to "promote" your URI to be more textually comprehensible, at the expense of losing some information about the input bytes, you can do `URI.asIRI()`, which will, to the extent possible, decode all percent-encodings and de-UTF-8 all the resulting octets.  The resulting URI is also textual, but follows the rules for encoding of non-ASCII characters.

Similarly, if you have some text a user typed and you need to ensure that it is ASCII-only for network serialization, you can call `.asURI()` on it.

`.asIRI()` has the subtle nuance of decoding any %-encoded ASCII-only characters as well, including the _extra_ subtle nuance of changing the _internal_ representation of `%2F` to `/`, so that applications can access completely decoded text in each segment.  However, since `/` cannot be serialized, `.asText()` will still %-encode slashes on the fly (without modifying the internal state of the `URL` being serialized; `URL` objects are completely immutable).

This is actually a restatement and expansion of the final part of the class docstring, and the rule is very simple, but its implications can be slightly confusing.  See for example this (ultimately invalid) bug report, which goes into some more detail: https://twistedmatrix.com/trac/ticket/8178


> Thanks @glyph. I'll try to play around with this at some point and see if it's useful for requests.

Thanks for considering this code to solve this problem, @Lukasa.  I'm optimistic you'll like it; we've been screwing up URLs real bad for a real long time in Twisted, and the design on this one really clicked together nicely.  The one part that might seem a little odd to Python programmers at first is the totally functional interface, but being able to treat URLs as immutable like the strings they came from actually ends up being really handy.


Oh and of course be sure to check out the [`click`](https://twistedmatrix.com/documents/15.5.0/api/twisted.python.url.URL.html#click) method, which handles relative reference resolution.  `URL` objects can represent both absolute and relative URLs, as well as non-HTTP URLs.  One possible feature that would be necessary would be relative reference resolution on already-parsed `URL`s, as well as just strings, which would be a minor change.


Looking back earlier in the discussion we may also be missing one other thing, which is an explicit relative-dot-resolution phase.  Currently this is just handled in `.click`, and there's an internal utility function for it.  It seems like we will need a `.normalizePath()` method or somesuch to accommodate that use-case fully.


Yeah, `.normalizePath()` is pretty important to this use-case. =)


@Lukasa `URL.normalizePath() → URL with no "." or ".." segments` work as a signature? Any other options which might be required? I could probably have that landed on trunk by Friday.


I think strictly we want to follow section 5 of RFC 3986. Note also that @sigmavirus24 has a library with exactly that name that may also end up a good fit here, so you may not want to leap immediately into writing that code: we need to work out exactly what we need here!


Looks like path normalization should follow [RFC algorithm](http://tools.ietf.org/html/rfc3986#section-5.2.4):

> 5.2.4.  Remove Dot Segments
> 
>    The pseudocode also refers to a "remove_dot_segments" routine for
>    interpreting and removing the special "." and ".." complete path
>    segments from a referenced path.  This is done after the path is
>    extracted from a reference, whether or not the path was relative, in
>    order to remove any invalid or extraneous dot-segments prior to
>    forming the target URI.  Although there are many ways to accomplish
>    this removal process, we describe a simple method using two string
>    buffers.
> 1.  The input buffer is initialized with the now-appended path
>       components and the output buffer is initialized to the empty
>       string.
> 2.  While the input buffer is not empty, loop as follows:
>    
>       A.  If the input buffer begins with a prefix of "../" or "./",
>           then remove that prefix from the input buffer; otherwise,
>    
>       B.  if the input buffer begins with a prefix of "/./" or "/.",
>           where "." is a complete path segment, then replace that
>           prefix with "/" in the input buffer; otherwise,
>    
>       C.  if the input buffer begins with a prefix of "/../" or "/..",
>           where ".." is a complete path segment, then replace that
>           prefix with "/" in the input buffer and remove the last
>           segment and its preceding "/" (if any) from the output
>           buffer; otherwise,
>    
>       D.  if the input buffer consists only of "." or "..", then remove
>           that from the input buffer; otherwise,
>    
>       E.  move the first path segment in the input buffer to the end of
>           the output buffer, including the initial "/" character (if
>           any) and any subsequent characters up to, but not including,
>           the next "/" character or the end of the input buffer.
> 3.  Finally, the output buffer is returned as the result of
>       remove_dot_segments.


We already have an implementation of exactly that algorithm (there's an RFC citation in the docstring and everything) it's just incompletely applied.  If you call `.click` with a proper relative reference, all those rules will be followed.  (If you call it with an absolute URI then you just get the absolute URI back with no dot segment removal; that might be a bug.)


I think it _is_ an issue because of this part of [5.2.2](http://tools.ietf.org/html/rfc3986#section-5.2.2):

`R` is relative and `T` is base. So here If `R` is absolute (i.e. has scheme and authority), its path should be normalized anyway

```
      if defined(R.scheme) then
         T.scheme    = R.scheme;
         T.authority = R.authority;
         T.path      = remove_dot_segments(R.path);
         T.query     = R.query;
```


Hi,
Rfc3986, section 5.4.2 mentions the exact case presented in this issue and states that every uri parser should handle those cases consistently.
I'm new around here, by the way, so, hello!


Testing rfc3986, shows that this already works there @lukasa and we had been talking about using that in the past. @glyph maybe we should try to converge our libraries