Raibek/requests-github-issues huggingface dataset:
URI encoding fixes

Fixes for Issue #369, as described in my [comment on that thread](https://github.com/kennethreitz/requests/issues/369#issuecomment-3953802).


Hey,

Can someone please compare this with our hotfix to see which is the better method to use:

https://github.com/kennethreitz/requests/issues/429

Thanks

Cal


@kennethreitz Thanks for merging. I forgot to add my name to AUTHORS. Could you please do that (Matt Giuca)?
@foxx I don't think we are talking about the same issue. I'll respond on the talk page for Issue #429.


@mgiuca ah - i see what you mean, looks like it is a completely different issue, thanks!


Hey say if I have a param like `{"percentage" : "35%"}`, how can I avoid requests to encode the percentage sign in my string? Thanks!


@dorafmon Why would you want to avoid requests encoding the percentage sign? That URL is not valid: there's a trailing percentage sign in it that _must_ be encoded.

However, if you really must have that, you should use the [prepared request flow](http://docs.python-requests.org/en/latest/user/advanced/#prepared-requests) and edit `prepped.url` to look the way you'd like it to.


@Lukasa , because some website do have a percentage sign in their get parameters so I kind of need to avoid the percentage sign to be encoded.


@dorafmon Unfortunately those websites are broken. There's no requirement for us to support that in the general API, so you'll need to use the prepared request flow as linked above.
Issue #335: Accept unusual keys in cookies.

Fixed a bug: RFC 2109 forbids certain characters, including colons and braces, from being in the key field of a cookie. Requests is compliant with RFC 2109, but many web frameworks are not. When requests receives such a key, a CookieError is raised and execution halts. This fix allows requests to accept and send such non-standards-compliant cookie keys.


Hah, the whole reason oreos exists is to add `[]_` to that list :)


Thanks, this looks great.
Unicode header dict values cause unhandled exception

Hey,

The following problem happens when you use a unicode string for a header dict:

```
_headers = {
    'test' : u'hello world'
}

requests.get('http://example.com', headers = _headers)

  File "/home/foxx/dev/iz/webapp/../webapp/libs/_requests/utils.py", line 152, in header_expand
    for i, (value, params) in enumerate(headers):
ValueError: need more than 1 value to unpack
```

This is due to the following code:

```

def header_expand(headers):
    """Returns an HTTP Header value string from a dictionary.

    Example expansion::

        {'text/x-dvi': {'q': '.8', 'mxb': '100000', 'mxt': '5.0'}, 'text/x-c': {}}
        # Accept: text/x-dvi; q=.8; mxb=100000; mxt=5.0, text/x-c

        (('text/x-dvi', {'q': '.8', 'mxb': '100000', 'mxt': '5.0'}), ('text/x-c', {}))
        # Accept: text/x-dvi; q=.8; mxb=100000; mxt=5.0, text/x-c
    """

    collector = []

    if isinstance(headers, dict):
        headers = list(headers.items())

    elif isinstance(headers, str):
        return headers
```

We have hotfix'd this by doing:

```
    elif isinstance(headers, str) or isinstance(headers, unicode):
        return headers
```

Hope this helps!

Cal


This is working for me in trunk:

```
[shivaram@good-fortune ~/workspace/requests]$ cat header_test.py 
import requests

_headers = {
    'test' : u'hello world'
}

r = requests.get('http://httpbin.org/headers', headers = _headers)

print r.content
[shivaram@good-fortune ~/workspace/requests]$ python ./header_test.py 
{
  "headers": {
    "Content-Length": "", 
    "Accept-Encoding": "identity, deflate, compress, gzip", 
    "Connection": "keep-alive", 
    "Accept": "*/*", 
    "User-Agent": "python-requests/0.12.0", 
    "Host": "httpbin.org", 
    "Test": "hello world", 
    "Content-Type": ""
  }
}
```

@foxx check your use case, see if this is closeable?
Cast the request URL as a string before using it

It would be great if requests would call `unicode()` on the URL parameter before using it, so that users can pass things masquerading as URL-strings, like an object from the excellent and extremely useful library [furl](https://github.com/gruns/furl).

This would turn this code:

```
requests.get(furl('http://example.com/ ').set({'query': 'string'}).url)
```

Into this:

```
requests.get(furl('http://example.com/ ').set({'query': 'string'}))
```

Which currently raises `AttributeError: 'furl' object has no attribute 'find'`.

Also, as a side note, it would be cool if you could use schema-less URLs and requests would default to http://

Edit: `unicode(furl_obj)` or `str(furl_obj)` is the same as `furl_obj.url`, by the way. That's what I mean when I say it's "masquerading" as a string.


This would be fantastic. 


As far you know, is it as simple as wrapping the URL argument with `unicode()`?


I think this would get hairy in python3 :)


I do welcome this change.


Resolved by #766.
unicode strings not allowed in headers

```
>>> import requests
>>> requests.get('www.google.com', headers={'foo': u'bar'})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/peter/Code/envs/test/lib/python2.7/site-packages/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/Users/peter/Code/envs/test/lib/python2.7/site-packages/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Users/peter/Code/envs/test/lib/python2.7/site-packages/requests/sessions.py", line 159, in request
    headers[k] = header_expand(v)
  File "/Users/peter/Code/envs/test/lib/python2.7/site-packages/requests/utils.py", line 152, in header_expand
    for i, (value, params) in enumerate(headers):
ValueError: need more than 1 value to unpack
```


We have a failing test for this right now. Working on it ;)


Contributions welcome, of course :)


I forked requests and was going to fix this but it appears that the latest checkout of the develop branch does not have this problem. I was using requests 0.10.1 from PyPI.


Dupe of https://github.com/kennethreitz/requests/issues/400


Edited :)
request.get('www.httpbin.org') throws error

## In [2]: r = requests.get("www.httpbin.org")

ValueError                                Traceback (most recent call last)
/home/kracekumar/<ipython-input-2-4bfef4bceb42> in <module>()
----> 1 r = requests.get("www.httpbin.org")

/usr/local/lib/python2.7/dist-packages/requests/api.pyc in get(url, *_kwargs)
     78 
     79     kwargs.setdefault('allow_redirects', True)
---> 80     return request('get', url, *_kwargs)
     81 
     82 

/usr/local/lib/python2.7/dist-packages/requests/api.pyc in request(method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, return_response, prefetch, session, config)
     65         return_response=return_response,
     66         config=config,
---> 67         prefetch=prefetch
     68     )
     69 

/usr/local/lib/python2.7/dist-packages/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, return_response, config, prefetch)
    189 
    190         # Send the HTTP Request.

--> 191         r.send(prefetch=prefetch)
    192 
    193         # Send any cookies back up the to the session.

/usr/local/lib/python2.7/dist-packages/requests/models.pyc in send(self, anyway, prefetch)
    345 
    346         # Build the URL

--> 347         url = self.full_url
    348 
    349         # Logging

/usr/local/lib/python2.7/dist-packages/requests/models.pyc in full_url(self)
    288 
    289         if not scheme:
--> 290             raise ValueError()
    291 
    292         netloc = netloc.encode('idna')

ValueError: 

This is kinda frustrating since `protocol` isn't menthioned underlying `urllib` is throwing error. So it would be awesome if `www.xyz.com` is converted to `http://www.xyz.com`. 


That's not a valid URL.


Browsers had made people to forget valid URL, everyone thinks www.xyz.com is valid. 


Developers don't.


oops. Then I aint look one :(
get with HTTPDigestAuth is returning 401 in 12.1 [EDIT: more data, updated requests]]

Although I can get HTTPDigestAuth to work on some servers, I have a case where it doesn't seem to trap the 401 and apply the credentials. I should state for the record that this all works in the browser. 

The server should reply with a 301 to another page, but it doesn't seem to be getting that far.

```
$ python
>>> import requests
>>> from requests.auth import HTTPDigestAuth
>>> r = requests.get("http://[server]", auth=('[user]','[pass]'), allow_redirects=True)
>>> r
<Response [401]>
>>> r.headers
r.headers
```

{'content-length': '349', 'content-encoding': 'gzip', 'date': 'Fri, 18 May 2012 21:29:32 GMT', 'content-type': 'text/html; charset=iso-8859-1', 'vary': 'Accept-Encoding', 'www-authenticate': 'Digest realm="[realm]", nonce="jniCP1bABAA=d312512776a608d2ebaa0d1a70af1b7a28b1c018", algorithm=MD5, domain="[server-realm]", qop="auth"', 'server': 'Apache/2.2.17 (Ubuntu)'}

Those headers look to me as though requests is not trapping the 401, and is just returning the server's initial response. Is there a header missing that requests is looking for? 


Aha! Requests' 401 hook is grabbing the 301 the server is returning -- and clobbering it by treating it like another 401.

When I send the request in such a way that no redirect is required, it works as advertised. 

Verbose logging makes it clear that the 301 is trapped and handled, but as the current code runs, the 401 handler never gets the chance to redo its hashes.

This seems like a bug to me.
Zlib decompress fix

#336 fixed.


Can't this argument to `decompress` be somehow calculated before call?


We shouldn't be relying on urllib3 for decompression right now, actually.


The fact that we are is a bug.


> We shouldn't be relying on urllib3 for decompression right now, actually.
> The fact that we are is a bug.

@kennethreitz I would agree. FWIW, I've come across some issues using requests (urllib3) with some gzipped responses.


This is a new bug that was introduced temporarily to fix a bug with our decompression. I'll use urllib3's deflate functions, but from requests.


@kennethreitz Any reasonable guesses as to when it might be squashed? Thanks very much.


Fixed urllib3 bug. Decided to always use stream decoding.
Make requests.request data kwarg accept file-like objects

It would be of great help to boto and others dealing with large files if requests.request's data kwarg accepted file-like objects, in addition to bytes and dicts. In the case of boto and S3, the max file upload size is 5GB. Many VMs and lesser machines don't have that much RAM, so this quickly grinds them to a halt. Being able to gradually read and upload would alleviate this issue.


@garnaat mentioning you here so you're aware of this issue.


why not use parameter `files`?

`:param files: (optional) Dictionary of 'filename': file-like-objects for multipart encoding upload.`


Because we're not dealing with multi-part uploads, if I understand correctly.

Also, @kennethreitz mentioned wanting to add generator support at the same time, so the 'data' kwarg doesn't care what you feed it, as long as you give it something. Maybe it's a file, maybe it's a huge string, it probably shouldn't care.
Redirect to unicode URL fails.

When I use requests 0.10.1 to get a URL like the following

http://www.amazon.co.uk/dp/B0002862UW

with a special user agent so that amazon redirects to a URL with unicode in it, requests will fail with a TooManyRedirects exception. E.g.

`r = requests.get(u'http://www.amazon.co.uk/dp/B0002862UW', headers={'User-Agent': 'Mozilla/5.0 (compatible; somebot)'})`


Thanks for the report. I'll try to push out a fix in the next release.


done!


Thank you so much!

On Wednesday, 15 February 2012 at 06:14, Kenneth Reitz wrote:

> done!
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/issues/414#issuecomment-3975549
added use_2to3=True to setup.


Why did you do this?


Sorry. I want to make the package 3.x compatible. use_2to3 is a keyword parameter added by Distribute. If use_2to3=True was set, setup script automatically apply 2to3 to package.

But, the code that I commit include a bug. Due to use_2to3=True, setup() in my commit will apply 2to3 even if python version is 2.x.

I should do like the following:

extras = {}
if is_py3:
    required.append('chardet2')
    extras.update(use_2to3=True)
else:
    required.append('chardet>=1.0.0')
    packages.append('requests.packages.oreos')

So, Could you please reject my pull request ?

regards.


It already is python3 compatible. No need to use 2to3.


The core of requests is 3.x compatible. But oreos and test_requests.py are not compatible.
Because of that, I misunderstood.
Indicated version on python-requests.org higher than what's actually released

[python-requests.org](http://docs.python-requests.org/en/latest/index.html) shows 0.10.2, but neither [PyPI](http://pypi.python.org/pypi/requests/) nor gits tags know about that release.


It's the development docs of the next release. 


But that means these docs are not for the version I get when following the installation instructions right next to the version number (well, at least not if I use pip or easy_install). That's not really elegant, is it?


Usually the overlap is only for a few hours. I'm just stuck on a few bugs right now.

If there was anything that changed usage, it'd say (dev), and the default would be the latest release.


Fair enough.


I agree that it's not ideal.
Fix SyntaxError on Python3 because "except Error, e" is not supported anymore

Since requests 0.10.1 dropped Python5 support, to catch an exception and
store the exception object in a variable the "as" keyword can be used so
Python3 will not raise a SyntaxError.

Kind regards,
Daniele Tricoli


s/Python5/Python2.5/ of course!

Cheers,
Daniele Tricoli


Oreos isn't imported or included in python 3. I'll merge anyway though :)
Requests no longer python2.5 compatible

Looks like this commit took it down 

https://github.com/kennethreitz/requests/commit/b0e2e4f284f619536b7f405122bb1a93900cd743

and specificially the use of the byte literal here
-    L = [chr(random.randrange(0, 256)).encode('utf-8') for i in range(n)]
  
  189     
-    return b"".join(L)


There are many commits that took it down — it was an intentional decision ;)


fair enough, you may want to take 2.5 off the listof supported versions in the docs so


Where do you see it?


 http://docs.python-requests.org/en/latest/community/faq/ in the python 3 support section


Whoops, I thought I fixed that :)
Cannot upload binary data using auth

Simplest way to reproduce that: 

```
requests.get('http://example.com/', auth=('a','b'), data='\xff')
…
  File "/usr/lib/python2.7/httplib.py", line 810, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)
```

It appear that when you specify auth parameters whole HTTP headers section becomes unicode string, and when httplib tries to append binary body to it the thing goes kaboom.

Probably this is caused by following lines in auth.py:

```
from __future__ import unicode_literals
```

and decode('utf8') in

```
return 'Basic ' + b64encode(("%s:%s" % (username, password)).encode('utf-8')).strip().decode('utf8')
```

However fixing this line isn't enough to make this work through proxy. This will still throw then:

```
requests.get('http://example.com/', auth=('a','b'), data='\xff', proxies=proxy_config)
```


I don't think that's the issue. It seems there's an issue with uploading binary data in general in 0.10.1. Working on it ;)


+1 for this problem. I will downgrade the requests if this is related only to 0.10.1.


Yes, It works in version 0.10.0.
Ability to override quote per request / session

Amazon S3 allows serving images with signed URLs. Such URL looks like:

https://BUCKET.s3.amazonaws.com/IMAGE?AWSAccessKeyId=KEY&Expires=EXPIRES&Signature=SIGNATURE

So site should calculate SIGNATURE for given URL and expiration time. Then I can fetch that URL using python-requests.

But the problem is that some languages (in my case PHP) does not properly quote URL (in my case PHP does not quote ":"). So while PHP site signs URL like "/some:image:name" python-requests tries to get URL like "/some%3Aimage%3Aname" and S3 refuses to proivde access: 

```
<Error><Code>SignatureDoesNotMatch</Code></Error>
```

That happens because PHP and Python encode URLs in different ways. And it would be nice if we have opportunity to override quote() per session. It may look like:

```
class PhpCompatSession(Session):
    def quote(self, url):
        ....

requests.get('URL', session=PhpCompatSession)
```

I can submit a patch if this proposal sounds good for you.


Related: issue #392 colon in url path is escaped


I still think this is a good idea, but it should be unnecessary now.
POST request without data payload leads to 411

This behavior was observed while using the requests.post() function on a cherrypy server. The response from the server: HTTP error 411 : no content-length specified.
When the data parameter was passed into the requests.post() function, issue was resolved. 


It would be nice if you wrote what version was used...
Also, [searching](https://github.com/kennethreitz/requests/issues/search?q=length) for length in opened issues brings issue #223 "Content-Length is missing" as the first hit.


Same here #356 & #223


Closing for #223
Redirection problem

Making a GET request with the python module Requests ends up with strange url:

``` python
>>> import requests
>>> r = requests.get("http://t.co/Uspy071j")
>>> print r.url
http://feeds.feedburner.com/%257Er/LesArdoises/%257E3/bD2JuJagz5I/roxino-cest-tout-vert.html?utm_source=twitterfeed&utm_medium=twitter
```

This url ends with an error 400. But using RestKit for the same url, the final_url return the correct value:

``` python
>>> import restkit
>>> r = restkit.request("http://t.co/Uspy071j", follow_redirect=True)
>>> print r.final_url
http://lesardoises.com/6277/roxino-cest-tout-vert.html?utm_medium=twitter&utm_source=twitterfeed
```


Well... suits this issue right to be #404.


Probably due to issue #269 fix subsequent redirect request method type


@yuvadm Yep, this is really a deal breaker for me. Its a shame 'cause requests fill all my needs but this redirection problem break the tool...

@piotr-dobrogost I hope this will be fixed soon, #269 was open 3 months ago...


@piotr-dobrogost I don't see how this would be related to that at all.


Looks like a result of a URL-escaping bug.


My mistake - it's GET not POST and has nothing to do with issue 269.


I think this has to do with #392.
From my experimentation it's the tilde.


It works now with the version 0.10.2
Added ability to check if missing trailing slash in path.

Previously this URL would not work:

```
http://example.com?foo=bar
```

But this URL would work:

```
http://example.com/?foo=bar
```

Even though a slash is required for a valid URL, it is a common use case
for the slash to be left out, so it would be a good idea to account for this.


A supporting reason to consider this patch...

An example of a missing slash can actually been seen on the Requests documentation homepage at http://docs.python-requests.org/en/latest/index.html :

```
r = requests.get('https://api.github.com', auth=('user', 'pass'))
```

If that example had any params such as the following, it would fail:

```
r = requests.get('https://api.github.com?foo=bar', auth=('user', 'pass'))
r = requests.get('https://api.github.com', auth=('user', 'pass'), params={'foo':'bar'})
```


Thanks!
Fix for #239: “Too many open files” 

The underlying problem appears to have been consuming async replies too late in two places:
1. async.send(prefetch=True) did not pass prefetch=True to the request.send() call but instead consumed the responses _after_ all of them had been processed, leading to an error any time the total number of requests exceeded the number of available file descriptors. acdha@495646bb6a89802a60208c2ab09749edba1f8e51 changes async.send to call request.send with the prefetch value.
2. Following redirects would result in only the last request in the chain being consumed, leading at least one file descriptor. acdha@67f50a0fe53765987e5a6a1c02daabe6400322af consumes each redirect response before following it.


Beautiful.


I might revisit this — there's really no need to download the content at all.


I'm not sure how much difference it will make for redirects - are you receiving non-zero byte contents from many servers? I'm assuming that if it fits in one 1500 byte packet it's probably not worth worrying about. I did verify that we need more than simply calling `raw.release_conn()` to actually close the socket but didn't investigate further.


Well the socket _should_ be kept open because it's within a PoolManager within a session. The connection itself should be closed though.


Perhaps sessions are leaving connections open when they should be killed. @shazow ?


Does this problem still exist in urllib3 1.2? The 'use poll instead of select' might fix this problem.

Can someone write code to replicate this issue in urllib3? I'll have a look then.


I believe that select is related. Unfortunately, poll is all we have on osx :(
Ability to declare how the content should be parsed

Just an idea. Make it super-simple to get the objects to users. When playing in the Python prompt with a new API, this looks pretty normal:

``` python
>>> import json
>>> import requests
>>> res = requests.get('http://example.com/data.json')
>>> data = json.loads(res.content)
...
```

Here's a suggestion that might add a little convenience for this kind of explorative work:

``` python
>>> import requests
>>> res = requests.get("http://example.com/data.json", as='json')
>>> data = res.content
...
```

The idea is, dynamically import a parser. There are reasonable defaults in the stdilb. It will only save a few lines of code, but would make it even easier to play around with APIs in the Python prompt. I thought the `as` option could accept `xml`, `json`, `text`, `csv`, `tsv` and `infer`. `infer` would use an appropriate the `Content-Type` and/or the file extension and/or `format=xml` in the URL to make the choice/

Possibly a more sensible way to approach this kind of thing would be to specify a callback:

``` python
requests.get("http://example.com/data.json", parse_content_with=json.loads)
```


This has been deeply considered before and decided against.

I definitely see the use case and understand the desire for this functionality, but it would quickly get out of scope. At the end of the day, requests is a HTTP module, not a HTML, JSON, YAML, XML, etc module. I'll link to a few past discussions below.


#151


#330


#310


Sorry for not looking into the backlog!


No worries at all, there are over 400 issues :)


I realize this is closed but just a thought: Is there really any benefit in...

``` python
requests.get("http://example.com/data.json", parse_content_with=json.loads)
```

Over...

``` python
json.loads(requests.get("http://example.com/data.json").content)
```

The latter seems more explicit and less magical. In fact if a requests response object was file-like or string-like, you could just do:

``` python
json.loads(requests.get("http://example.com/data.json"))
```


There is not.


:cake:
Track upload/download progress

I think this would be an interesting feature for Requests.
I really miss it!


Moved to [Urllib3](https://github.com/shazow/urllib3/issues/43).


Use the content-length header to find out the size of the content being downloaded, and then use [iter_content](http://docs.python-requests.org/en/latest/api/?highlight=iter#requests.Response.iter_content) to download it, updating your download progress indicator after each loop.


What @johtso said :)

We should add this to the docs. People ask for it so often :)


@kennethreitz And for upload?


Streaming uploads aren't supported at this time.


As @johtso said, this is very easy with `iter_content`. Add this code sample to the documentation and close this issue?

```
import requests
from clint.textui import progress

CHUNK_SIZE = 1024 * 10

response = requests.get('http://files-r-us.com/bigfile.tar')
content_length = response.headers.get('content-length')
iter_content = response.iter_content(CHUNK_SIZE)

# Display progress bar if content-length is set
if content_length is not None:
    expected_ittimes = int(content_length) / CHUNK_SIZE
    iter_content = progress.bar(iter_content, expected_size=expected_ittimes)

with tempfile.NamedTemporaryFile(prefix='song-', suffix=extension, delete=False) \
         as fp:
    for chunk in iter_content:
        fp.write(chunk)
```


Sorry, you forgot a part of the issue. Upload progress tracking.


The issue is closed on urllib3 :disappointed:
Quoted urls don't work in newer versions

This worked in 9.1
requests.get('http://knowyourmeme.com/memes/ming-general-sushi-%E6%98%8E%E5%B0%87%E5%A3%BD%E5%8F%B8/photos').status_code

And would give 200 as a status code.  In 10.1 it doesn't work.  I also tried to unquote it and utf-8 decode it and neither of those worked.  I see that several unicode changes happened so it may just be that this was intentional but I am not sure how such a url should be provided then.


This was fixed in #387. It works with current develop branch.


200 again!
Support proxies set in environment variables

A common standard supported by many command-line tools is to use environment variables to configure proxies. The variables are described in full here:

http://www.gnu.org/software/wget/manual/html_node/Proxies.html

According to the documentation, urllib and urllib2 support this.

Note that the environment variable names for these settings are considered to be case-insensitive, so `http_proxy` or `HTTP_PROXY` should work.

I'm happy to have a go at implementing this, if you can point me to the correct place in the code where you think it should be handled. Also, is there a guide anywhere detailing how to run the test suite?

Cheers


I would _love_ this.


Are you sure this is not working already - I tested using

``` sh
pip install mitmproxy
mitmproxy
```

This works:

``` sh
export https_proxy=http://localhost:8080
python -c "import requests; requests.get('https://github.com/')"
```

This doesn't

``` sh
export https_proxy=http://localhost:8081
python -c "import requests; requests.get('https://github.com/')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "requests/api.py", line 54, in get
    return request('get', url, **kwargs)
  File "requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 605, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: https://github.com/
```


Commit was SHA: 0ba8c442


This was done forever ago :)


:thumbsup: thanks!
colon in url path is escaped

if you get a url with this format http://www.colon.com/kql:a/in, the colon after kql becomes escaped and as far as i understood it shouldn't.

Same behavior does not happen in urllib2.

See http://stackoverflow.com/questions/2053132/is-a-colon-safe-for-friendly-url-use


The question you refer to talks specifically about [fragment part](http://tools.ietf.org/html/rfc3986#section-3.5) (the part of url after `#` sign). The example you show has `:` sign in the **path** part. However, according to [3.3. Path](http://tools.ietf.org/html/rfc3986#section-3.3) section of RFC3986 `:` is allowed in this part of url as well. I'm not sure if some HTML specs are more strict about this (see answer of irreputable to the SO question).


After having a good read of the article it seems to me that the ":" should not be escaped. Firefox and urllib2 do not escape it so i don't understand why this library should


If you had also a good read of my post here you would know that I **agree** with you :)
Btw, this calls for new `url` endpoint in httpbin I guess :)


I'd like to add a configuration for disabling url escaping.


I'm going to fix this.


Thanks guys :)


The same issue with `=` sign in the path. For example, try to `get` this URL: `http://lh3.ggpht.com/pQVGhZF5dQJmCaJqiKX6jZx6cKXJ1DL-PQkrZxtkelPPrXaIRIZK2t1t-uCbsKAIFbA=w124`, and you'll get 404 instead of the image.


I think I have the same issue with `~`:
`http://feedproxy.google.com/~r/StephenHoliday/~3/E64PhsZa2_w/fidofetch-architecture`


As of requests 0.10.2, the colon is actually escaped twice. The first escaping happens at utils.requote_path(), which is called from Request.full_url. The second escaping happens in Request.path_url method: 

path = quote(path.encode('utf-8'))

As a result, it's not possible to send a valid request containing a colon in the path part.


`encode_uri` configuration was added!


Kenneth,
I see absolutely no reason why this behavior should be configurable. The problems currently experienced by the users are caused by double escaping (see my comment above), which is simply a bug and needs to be fixed.


@yole there are always edge cases.
re-enable support for scalar values in headers dict

Between 0.8.x and 0.10.x, support for this was dropped:

requests.get(resource, headers={'Authorization': 'string'})

It now requires a dict as the header value. Can we get support for both dict and string as the value?


Never mind, looks like I was passing a dict as the header value (expecting it to be automagically json dumped), and getting confused by the error. Sorry about the noise.


No worries, I'm glad you got it figured out ;)


FYI, I came back to this and realized it was because of the problem this fixed: https://github.com/kislyuk/requests/commit/f8b3141204c83737fd6ab7a5ba3aa84e0a1b6d61

Your latest tag (0.10.1) still contains that error, do you mind tagging 0.10.2 with this fix?

Thanks.
add failing test for unicode headers (python 2.x only)

This adds a failing test for unicode headers in python 2.x (I've tested python 2.6) 
I've tried to fix this the original problem is that in [utils.py (line 149)](https://github.com/kennethreitz/requests/blob/develop/requests/utils.py#L149) doesn't check for unicode, or rather that it doesn't import the str and bytes definitions from compat). However, you _might_ want to change the order of the [definitions of str and bytes for python 2 in compat](https://github.com/kennethreitz/requests/blob/develop/requests/compat.py#L89), because, at the moment, they end up both being defined as unicode.
Right, so, if one fixes this (I've got the code if you want that too), and then there ends up being a problem with gevent's socket just calling data.decode(), which fails on my linux box, because that ends up trying to convert everything to ascii (which seems to be default behaviour).
I can work around this by making sure I only ever pass py2's strs as keys and values, but I'd like to make it work if you can point me in the right direction.


Thanks a lot! We'll have to fix this.
Develop

I noticed yesterday that requests doesn't work on Python 2.5 ( or Jython ). These commits make running the tests slightly easier to setup.

The core tests now pass on cPython 2.5. All tests (core, async, ext) pass on cPython 2.6. Working on changes to support Jython (currently one failing test in test_safe_mode).

If you want me to rework anything, let me know. But I really would like to re-enable Python 2.5 support.


I appreciate the contribution, but Python 2.5 support was dropped intentionally. 

While it is fairly simple to support again (as this pull request shows), It's a bit more subtle than that. The main issue is that `except RequestException, e` is invalid syntax in Python 3. I'd also like to move to modern string formatting, which isn't possible when supporting 2.5.


I think the idea of having different requirements files is nice and worth merging. Btw, why is nose a requirement?

@seanjensengrey You say all tests including new async ones pass. I'm very surprised. See issue #383 Any comments on this?


@kennethreitz What if I added in using the 2to3 into the Makefile so we could generate a Python 3 compatible install? The vast majority of users are still on the Python 2.x line and will be for the foreseeable future.

Requests is great and I would like to see it replace all the other cruft out there by being available everywhere. It would be nice if I could run it on cPython 2.5 and Jython, it limits my use of it on the systems I use as I can't migrate programs across environments.


For now, if you'd like Python2.5 support, using a version before 0.10 is probably best.
fix problem with path being double escaped

On python 2.6.4 (at least) the path portion is getting escaped twice -> once in Request.full_url and once in Request.path_url
This fixes it, but I don't know why there was a commented test for Python 3.


Nice catch. thanks!


Removing the check is not the right way to go. It fails exactly the same way in Py3. You have to remove the two lines to get it working there too.
Requests and urllib3 using one test platform?

Basically the needs of the two libs as far as test environment is concerned are the same. It looks like there would be much benefit if requests could reuse test platform provided by urllib3.


@shazow have (_very_ lightly) been discussing the idea of merging our projects in the mid-future. If that comes to fruition, that will be the time to merge test suites.

In the meantime, I'm fine with requests "trusting" that `urllib3` is tested and reliable.


> @shazow have (very lightly) been discussing the idea of merging our projects in the mid-future

That's really interesting.

> In the meantime, I'm fine with requests "trusting" that urllib3 is tested and reliable.

I was not implying it's not the case. I think it would be easier to have one test platform used by both libs. Taking into account that requests already embeds urllib3 there's no reason not to use its test platform if it supports all features requests' tests need. This way we could start taking advantage of this right away and in addition there would be less work to do when merging projects :)


On the other hand, it's kind of nice that we're not testing against the same webserver in the event that one of our's is buggy, at least there's some chance we'd catch the bug on the other layer. :P


@shazow +1


> On the other hand, it's kind of nice that we're not testing against the same webserver

Am I missing something or is test platform in urllib3 going to have multiple backends? If so then both libs would in fact have been tested even better than now, right?


@piotr-dobrogost Not sure what you mean by multiple backends.

Originally we had a wsgiref simple server, then we moved to an evenlet wsgi app server, now we're moving to a Tornado wsgi server and a socket-handler based thread server.


Oh, well I thought you're adding support for Tornado not switching to it.


All in the name of Python3 :)
Docs don't show how to use config API

Hey. Newb user here. I've just spent about 30 minutes figuring out how to use the config API to set 'danger_mode' = True.

The docs don't seem to show this (or at least, I couldn't find it)

I was trying all sorts of wrong things, based on what I saw in the docs under 'Configuration API Docs' (http://docs.python-requests.org/en/latest/api/#configurations), such as:

import requests
requests.defaults.danger_zone = True

In the end I resorted to looking in the source.

I think the docs need a short sentance and example of setting a config option.

Many thanks.


I was looking at config today and ran into the same issue. For the time being, requests.get('http://example.com').config gives a good summary of what config options are available. 


http://docs.python-requests.org/en/latest/user/advanced/#configuring-requests


I don't want to be unhelpful, but I followed the link to the docs and had a five minute search around, but I'm still completely in the dark as to how to use the config. If I had to do it again today, I'd have to go look at the source again to figure out where I pass in or modify a config dict. Am I being dumb? With respect, if the bug is 'wontfix', I can understand and that's cool, but for the record I don't think this documentation bug is fixed. Peace.
Standard tests from RequestsTestSuite run using async api

Almost all are failing... Need some feedback if it makes any sense.


I haven't tried running these yet, but at first-glance these look great.

I'm not a fan of the `session=s`, but I see the use case :)


@piotr-dobrogost I am seeing all the async tests pass under python 2.6, I am not able test under 2.5 right now as my gevent isn't installed properly.

```
nosetests --with-color test_requests_async.py
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 0.872 seconds
OK
```

Is that expected behavior?


Could you please run tests against http://httpbin.org - that's what I do and most tests fail.
Could you please also run below script a couple of times. I get about 1-3 `Max retries exceeded` errors when running this.

``` python
from gevent import monkey
monkey.patch_all()

import gevent
import requests

def http_req(i, url):
    for j in xrange(2):
        r = requests.get(url)
        print "Request ", i, j, r


if __name__ == '__main__':
    greens = [gevent.spawn(http_req, i, 'http://httpbin.org')
                for i in xrange(50)]
    print "waiting end of greenlets ..."
    gevent.joinall(greens)
```


Running your script I do see some failures that I don't see with the local tests.

```
Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "test_piotr.py", line 9, in http_req
    r = requests.get(url)
  File "/Users/s.jensengrey/wxz/python-requests/requests/requests/api.py", line 51, in get
    return request('get', url, **kwargs)
  File "/Users/s.jensengrey/wxz/python-requests/requests/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Users/s.jensengrey/wxz/python-requests/requests/requests/sessions.py", line 200, in request
    r.send(prefetch=prefetch)
  File "/Users/s.jensengrey/wxz/python-requests/requests/requests/models.py", line 537, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host=u'httpbin.org', port=80): Max retries exceeded with url: /
<Greenlet at 0x101227550: http_req(47, 'http://httpbin.org')> failed with ConnectionError
```

Not sure if that is a failure in `requests` itself or the server running on httpbin ?


I don't know either; urllib3 masks network errors - see shazow/urllib3#32.
Could you run async tests against httpbin.org?


I just did a 

```
HTTPBIN_URL=http://httpbin.org/ make test
```

and all tests pass (requests, requests_ext, requests_async) on my fork. I added in a print statement to confirm the test url. My local server was showing no hits during the test pass.
Moved dispatch_hook call for pre_request into send method

In the case of using requests.Request or async.request the pre_request hook wasn't very useful because it was called when the request was made rather than when the request was being sent. I moved the dispatch_hook call to just before the call out to conn.urlopen in request.send. Considering that multiple hooks was implemented instead of Response.response_time, I think that this is important because it allows users to time responses. Example: https://gist.github.com/1667527


Thanks, I like this!

Maybe it would be better to have a pre-pre-request hook too ;)


I considered that, but I didn't know how much value people would find in it. We already have the args hood being called wen the request is created.


Now that we can have multiple hooks, I'd like to use hooks internally more. Most of the processing steps could become hooks.


This actually moves it to a place where it is not anymore possible to prevent a http request like described in http://notebook.okfn.org/2011/10/08/implementing-an-http-cache-using-requests-api-hooks/

Please consider moving it a bit more higher :-)


This is a different use-case than what I use 'pre_request' for. Does this maybe warrant another hook? Maybe a 'creation' hook?


Ah. I forgot about the `if not self.sent or anyway:`. I think this change looks good
requests.async -- catch DNS errors more explicitly?

This is related to #380, however the manifestation is different when using requests.async

This could be handled by detected that the redirect URL is bogus.  On the other hand, it would be good to have a more explicit exception than an low-level IOError when gevent's DNS infrastructure gets confused.

What's the best way to fix this?

```
>>> r = requests.get(
...     'http://travel.rr.com/in?a=roadrunner&;p=heroluxuryvacation&;url=/dealssearchcat=Luxury&;rp=t&;nv=t&;pz=15&;pgsrc=roadrunner',
...     allow_redirects=False)
>>> 
>>> r
<Response [302]>
>>> r.headers.get('location')
'http://www.kayak.com?;nv=t&;pz=15&;url=%2Fdealssearchcat%3DLuxury&;pgsrc=roadrunner&;p=heroluxuryvacation&;rp=t'
>>> 
>>> 
>>> import requests.async
>>> r = requests.async.get('http://travel.rr.com/in?a=roadrunner&;p=heroluxuryvacation&;url=/dealssearchcat=Luxury&;rp=t&;nv=t&;pz=15&;pgsrc=roadrunner')
>>> r.send()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 514, in send
    self._build_response(r)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 253, in _build_response
    request.send()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 492, in send
    timeout=self.timeout,
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 336, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py", line 217, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "httplib.py", line 970, in request
    self._send_request(method, url, body, headers)
  File "httplib.py", line 1004, in _send_request
    self.endheaders(body)
  File "httplib.py", line 966, in endheaders
    self._send_output(message_body)
  File "httplib.py", line 826, in _send_output
    self.send(msg)
  File "httplib.py", line 779, in send
    self.connect()
  File "httplib.py", line 760, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 553, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/pymodules/python2.7/gevent/socket.py", line 681, in getaddrinfo
    _ttl, addrs = resolve_ipv4(host, evdns_flags)
  File "/usr/lib/pymodules/python2.7/gevent/dns.py", line 56, in resolve_ipv4
    core.dns_resolve_ipv4(name, flags, waiter.switch_args)
  File "evdns.pxi", line 93, in gevent.core.dns_resolve_ipv4 (gevent/core.c:6533)
IOError: evdns_resolve_ipv4('www.kayak.com?;nv=t&;pz=15&;url=%2Fdealssearchcat%3DLuxury&;pgsrc=roadrunner&;p=heroluxuryvacation&;rp=t', 0) returned -1
>>> 
```


You're right. We should be catching these gevent errors and mapping them to regular request exceptions.


Async is removed from requests now. A new module, [grequests](https://github.com/kennethreitz/grequests) will replace it.
Added missing parentheses and “self.”

Use e.g., http://telofy.spline.de/foo/lxml-bug-690110.html to test
the error handling. (Originally created for an lxml bug.)

Is the first block (ll. 751–755) even necessary?


Parens aren't needed there, but I'll take it :)


Thanks for pulling; great library you have here. Regarding the parentheses, I think they are. Otherwise any `UnicodeError` is assigned to `TypeError`, while a `TypeError` is not caught, leading to behavior that looks utterly mind-boggling at first glance:

```
In [5]: resp = requests.get('http://telofy.spline.de/foo/lxml-bug-690110.html')

In [6]: resp.text
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/home/telofy/documents/university/bsc/memaw/<ipython-input-6-89da6afc31f1> in <module>()
----> 1 resp.text

/home/telofy/.buildout/eggs/requests-0.10.0-py2.7.egg/requests/models.pyc in text(self)
    758         if not content:
    759             try:
--> 760                 content = unicode(content, encoding, errors='replace')
    761             except UnicodeError, TypeError:
    762                 pass

TypeError: coercing to Unicode: need string or buffer, NoneType found
```
async - RuntimeError: The content for this response was already consumed

``` python
from requests import async

request = async.get('http://httpbin.org/get')
response = async.map([request])[0]
print response, len(response.content)
```

OUTPUT

``` pytb
Traceback (most recent call last):
  File "C:\Users\Piotr\sandbox\python\requets_content_already_consumed.py", line 5, in <module>
    print response, len(response.content)
  File "c:\users\piotr\projects\requests\requests\models.py", line 719, in content
    'The content for this response was already consumed')
RuntimeError: The content for this response was already consumed
```

This error is **NON DETERMINISTIC**
Tested on Windows Vista x64 with Python 2.7.2 x64, gevent 0.13.6 x64, requests 0.10.0


I get the same error with requests 0.10.1, gevent 0.13.6, on ubuntu


I have this error when the server I'm trying to make a request to is not reachable.


I am seeing the same thing on centos with gevent 0.13.6.


I have been having troubles with this as well.

Looking at this, the problem is is that the [self._content is set to None when there is no content](https://github.com/kennethreitz/requests/blob/18591920f3599d34a8c50d371d58d0d33bd4623f/requests/models.py#L736) (for an error or something similar), however once the content has been checked the flag is set and then the next time someone looks at content they get this error. [Prefetch is set to true on the async modules](https://github.com/kennethreitz/requests/blob/18591920f3599d34a8c50d371d58d0d33bd4623f/requests/async.py#L71), so this error is more likely to be encountered on errors (someone checking content on a regular request will get None the first time and an error the subsequent times).

I am going to create a pull request that makes the _content attribute on response False on initialization and check if content is False now. 


Fixed thanks to @newmaniese
Instead of select, we should use poll.

select only goes up to 1024 FDs.  So once we have used more than that,
this would fail.  poll (or epoll) fixes this.  This solution uses poll
and seems to work.

My only test has been using a single session to fetch 10**4 pages a few
times.

Is there a more robust testing tool (framework?) that I should use.

PS:  Ken et al,  this is my first attempt at a pull request, so if this is not
the right way to do it, please educate me.  I have several more suggested
changes that I'd like to put forward for your consideration.


Because urllib3 is package from outside (http://code.google.com/p/urllib3/) I think you should push this patch to them directly.
Should standard tests be run in async mode - feedback needed

There's going to be new module test_requests_async with tests for requests.async module.

Should tests from current test_requests be somehow included in this new module so that each "standard" test would be run in async mode as well?
If so how to go about "reusing" these tests without coping them to the new test_requests_async (with changes needed due to the current difference between normal and async api - mainly the need to call map() in the latter case)?

Sample code in issue #383
which Gevent to install?

Requests.async relies on the Gevent package. 

Question: which Gevent should I install, 0.9.x or 1.x?

(The 0.9.x branch of Gevent relies on libevent, but 1.x is based on libev.
Does this have any consequence on requests.async?
## 

Pengkui


It shouldn't matter. The latest is recommended though :)