Raibek/requests-github-issues huggingface dataset:
Python 3: data dict is incorrectly encoded for multi-part file uploads

**This problem is Python 3 specific.** I am performing this test on Python 3.2.2 and `requests` 0.10.8.

Take a look at the following minimal example code.

``` python
import requests
import io

response = requests.post("http://www.joelverhagen.com/sandbox/tests/upload.php", data = {
    "foo": "bar",
    "jibba": "jabba",
})

print("-- WITHOUT UPLOAD --")
print(response.text)

exampleFile = io.StringIO("Hello, world.")
exampleFile.seek(0)

response = requests.post("http://www.joelverhagen.com/sandbox/tests/upload.php", data = {
    "foo": "bar",
    "jibba": "jabba",
}, files = {
    "file": ("file.txt", exampleFile)
})

print("-- WITH UPLOAD --")
print(response.text)
```

The script being queries is on my own website, and I will leave it up until this issue is closed. The output is the following:

```
-- WITHOUT UPLOAD --
Array
(
    [jibba] => jabba
    [foo] => bar
)

-- WITH UPLOAD --
Array
(
    [b'jibba'] => jabba
    [b'foo'] => bar
)
```

As you can see, the keys of the `data` keyword argument are not encoded correctly when there is a file upload. I have done some digging, and I have tracked down the problem. In **models.py**, there is a static method on the `Request` class called `_encode_params`. This encodes each of the values and keys to UTF-8 bytes using `.encode('utf-8')`. In Python 3, this results in the new `bytes` type, whereas in Python 2, a normal string was returned. 

This doesn't cause a problem until `urllib3`'s `encode_multipart_formdata` function (in **filepost.py**) uses the string formatting operator `%` and `%s` to concat the key into the request body. This basically ends up calling `str()` on the bytes object which is why we see the `b''` literal in the key name.

I wasn't sure whether this is a problem with `urllib3` or `requests` because I'm not sure if [shazow](https://github.com/shazow/urllib3/tree/master/urllib3) want bytes or strings being sent to `encode_multipart_formdata`.
potentially fixes #338

This attempts to fix an issue where encoding of a string might fail when the
encoding is set to some unknown format. Here we attempt to catch the
LookupException and subsequently blindly encode the string one final time.
That is we call str() over response.content without specifying an encoding.
This may still fail in certain cases but does properly handle the case of #338
by returning the expected string.


Thanks @piotr-dobrogost, I've updated with a (hopefully) better comment.


Love this. Thanks!
adding a request_status attribute to HTTPError

HTTPError provides a nice way to handle errors via response.raise_for_status,
however even though the HTTP status code is included in the error message
string, this does not necessarily provide a clean API with which to interface
over the request's status_code when an error occurs.

Particularly this is useful when using a module or class that wraps requests
and returns some parsed response but leaves error handling up to whatever
is using it. For instance, if such a class were to call raise_for_status,
while it's possible to see if a string representation of an HTTP code is in the
error message via str(404) in (str), this isn't as clean as being able to
simply use e.status_code and compare directly via 404 == e.status_code.


Hmm, I like it. I think it would be better to give it the entire response object though. Closing for now :)


Want me to update this and use the response object instead? :)


That'd be great! 


Also, you accidentally changed the exception messages to all be redirect :)


Damn, I was hoping you'd not notice my foolishness; I guess I didn't update the pull request in time. :( //shamed


hehe :)


:sparkles: :cake: :sparkles: 

Thanks!
Generate chunked ValueError fix

Here's a potential fix for #436 - @RonnyPfannschmidt is this what you had in mind?


You're the best!

The `t.co` test needs to be moved to `text_requests_ext.py`, because `test_requests.py` needs to be workable locally `


:+1:


Merged! I'll try to get this out later today.


:sparkles: :cake: :sparkles:


looks good


Awesome, thanks Kenneth - this is huge for us, this and the CookieError stuff are 100% of our requests-related errors!


Released! Cookie errors should be suppressed now too. Let me know if you have nay issues
Tox for testing

Hey there, maintainer dudes,

Keeping track of the setup environments for python 2 and 3 (and keeping them separate)
was driving me nuts, so I've added support for testing with tox and nosetests.

If you have tox installed--

```
 pip install tox
```

Then for each environment it will build requests into its own virtualenv with its own dependencies, 
run all the tests, and give you results.

Currently, this exposes that gevent is outside of the requests setup.py
(by design, I imagine) so it needs to be included seperately for
test_requests_async.py to work without throwing a RunTimeException.  

Some tests in the Python 3 environment fail because 

And envoy is not in the requirements of setup, despite being referenced in the test cases.

Tox is a great way to test on multiple virtual environments (and it interfaces with hudson jenkins too, if that counts as a win).

Danver Braganza


Hey there. I actually had tox setup for a while for requests, but prefer to make that a personal decision:

http://ci.kennethreitz.com/job/requests/

As long as the tox.ini should never require any configuration from the user, I may not be against adding this back in.


Holding off on this for now. I'll reconsider once I get the new test stuff in place (python3 really messed it up :)


I guess I just want to keep this out of source control. Maybe we could keep it in a gist and have the existing `make init` pull that down locally if needed.
Proxy configuration by environment variables

Hi,

I recently received [an issue](https://github.com/jgorset/facepy/issues/21) that describes difficulties in using proxies with [Facepy](http://github.com/jgorset/facepy), a Facebook Graph API client that uses requests internally. I would like to keep proxies out of my interface, so I've facilitated the configuration of proxies in requests through the environment variables `HTTP_PROXY` and `HTTPS_PROXY`.

I'm not sure how I might test this feature, but if you have any ideas please let me know and I'll get right to it.


You're incredible :)


Now, now... ;-)


:sparkles: :cake: :sparkles:
A fix for https://github.com/kennethreitz/requests/issues/461

Added testcase showing a non-standard unicode character (in this case,
A-umlaut), returned by the server in a redirect, breaks py27 requests.

I have a class of urls which I need to get in py27 which break,
because the server returns a _relative redirect_ with a high unicode
character in it.  Because of how str and unicode are re-bound in py_27
mode for requests, this means that unicode is being called on an
object that does not implicitly convert to unicode (it has an ord >
127).

The other minor change I made is a small hack to remove the unicode-literal u from
test_requests_ext.py so it can be run on python3.   

This fix, which seems to work for me (hurray), is to urllib.quote the
relative returned by the server.  This percent-encodes the non-standard
characters in the unicode, and then everything behaves nicely.

All tests pass for python 2.7.2 and python 3.2.2 on Linux.


Okay I seem to have made a real dog's dinner of this pull request.  By accident I pushed some commits relevant to testing to this branch.  I've now reverted these.  Herp.  Derp.

The truth is that for `requests.compat.urljoin()` to work, it needs to accept ANSI-7 compatible strings.  I found another test case that was breaking, and I've now fixed that too by using the extremely excellent `requests.util.requote_uri` instead of just `requests.compat.quote`.


All merged :)
Raise ValueError, not KeyError, in case the schema is unknown

Requests raises ValueError for most incorrect urls, but in case the schema is unknown (e.g. "foo://a/b/c"), KeyError is used.

```
>>> requests.get('foo://a/b/c')
KeyError: u'foo'

>>> requests.get('x//a/b/c')
ValueError: Invalid URL 'x//a/b/c': No schema supplied
```

This makes it hard to differentiate between "erroneous url" versus "http request failed".
Fix and considerably simplify the iter_lines() function

1. Do not use rstrip() as this would also throw away meaningful whitespaces
2. Avoid assuming what striplines() considers a line break terminator, the list of those is quite long in Unicode.
3. Add tests for the above edge cases.

This should look much better now. The only question I have is whether iter_content() could yield an empty chunk? generate()/generate_chunked() never do but the decoder might. In that case, I'll need to amend this.


Oh, and I removed the chunk_size argument as it doesn't make much sense (it has no effect whatsoever on the output of the generator).


Now that I think about it, this is wrong and the old code was correct when it comes to \r\n split across two chunks.


I love it!

Do you think chunk_size would never be needed on iter_lines?


I don't think so. It's arguably not very useful for iter_content() either as long it's approximate and doesn't reflect the post-decompression stream (my plan is to get to that one next). But for iter_lines, it really only fine-tunes how the function works and not its output so that should justify its removal unless I'm missing something...

I'll need to give this one more iteration to handle the multi-character line terminators properly.


@kennethreitz It's kind of strange that iter_lines() does not perform decoding (or even has an argument to control that). There is no concept of lines in binary streams. Any specific reason for that?


Yes it does, it has `decode_unicode`, which is false by default. You can't always infer encoding.


@kennethreitz Sure but it doesn't even try. Encoding detection only runs for text(). Lines are a text concept so iter_lines() should perform both the encoding detection step and actually decode. If it's impossible, it should ideally fail.


I think I'm going to merge this, adding back in chunk_size for backwards compatibility if anything else.


Gah, unfortunately this wasn't compatible with python3.


Thanks a million for this, btw. I hated that code :)
url is quoted twice before finally sending the request

well this is my code:

``` Python
import logging
logging.getLogger().setLevel(logging.DEBUG)
import requests
logging.info('requests version: %s' % requests.__version__)

requests.get('http://www.baidu.com/omg@requests.com')
```

and this is the output I got from terminal after running the code

> INFO:root:requests version: 0.10.1
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): www.baidu.com
> DEBUG:requests.packages.urllib3.connectionpool:"GET /omg%2540requests.com HTTP/1.1" 302 222
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (2): www.baidu.com
> DEBUG:requests.packages.urllib3.connectionpool:"GET /search/error.html HTTP/1.1" 200 1262

Look at the third line of results, what do you see?

```
/omg%2540requests.com
```

OMG, path of the url was quoted twice! (omg@requests.com -> omg%40requests.com -> omg%2540requests.com)

Although the request I made was just a sample and has no actual meaning, but this is a very serious BUG, I tried this url with Chrome, curl and wget, everyone of them works fine.

It's very curious that the newest version 0.10.6 doesn't have this problem, but I still want to grumble a little about it :\, cause requests, my favourite python request module, almost made me crazy this afternoon


Supporting Python3 caused a lot of bugs :)

Bugs that are now worked out with the new url quoting system.


LIke you said, it was a very serious bug, and it was fixed :)


@kennethreitz Yes guy, it has been fixed, and you are really efficient :)
requests.get response.content is no longer unicode in 0.10.6

Hi

this code:
<code>
import requests
response = requests.get("http://localhost:8888/", allow_redirects=False)
print type(response.content)
</code>

in requests=0.9.1 prints 

type 'unicode'

but now, with requests=0.10.6 prints

type 'str'

it's break the backward compatibility :(

http headers for this request:

< HTTP/1.1 200 OK
< Content-Length: 1627
< Etag: "0d4a9d4d7ccb27902918a63155d797ca80389ab9"
< Content-Type: text/html; charset=UTF-8
< Server: TornadoServer/2.0


Sorry... 


`response.content` is bytes. `response.text` is unicode.


Sorry about this, I just read the History file and find this warning.
Decode URL to utf-8 before joining.

To avoid UnicodeDecodeError's like on http://blip.fm/~1abvfu

Note: This doesn't have a test, which sucks, because for the life of me I can't seem to figure out how to trigger this error with httpbin. Maybe flask has better handling of UTF-8 characters than the server used at blip.fm and others? Really not sure.

Here's an ad-hoc test for you:

Before:

```
>>> import requests
>>> requests.head('http://blip.fm/~1abvfu', allow_redirects=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 73, in head
    return request('head', url, **kwargs)
  File "requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "requests/sessions.py", line 200, in request
    r.send(prefetch=prefetch)
  File "requests/models.py", line 559, in send
    self._build_response(r)
  File "requests/models.py", line 232, in _build_response
    url = urljoin(r.url, url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urlparse.py", line 219, in urljoin
    params, query, fragment))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urlparse.py", line 184, in urlunparse
    return urlunsplit((scheme, netloc, url, query, fragment))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urlparse.py", line 190, in urlunsplit
    url = '//' + (netloc or '') + url
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128)
```

After:

```
>>> import requests
>>> requests.head('http://blip.fm/~1abvfu', allow_redirects=True)
<Response [200]>
```


Note that you can't assume that the Location header will be UTF-8 (as opposed to another encoding), so the correct fix is actually to quote the string (the commit in #472 gets it right)


Glad to hear someone took care of it.
Path quoting is unnecessarily aggressive

`utils.requote_path` passes `safe=b""` into `urllib.quote`, which causes some characters to be encoded that don't have to be. This is allowed by the [RFC](http://www.faqs.org/rfcs/rfc1738.html), however there is some ambiguity in the RFC as to whether the encoded characters have the same meaning as their unencoded representations. This ambiguity allows [some web servers](http://www.latimes.com/health/la-he-qnexa-fda-20120223,0,1643846.story) to consider _/a,b_ and _/a%2Cb_ to refer to different resources. I think we should only encode characters that must be encoded, as that should lead to the widest compatibility. This would mean passing `safe=b"$+!*'(),"` to urllib.quote. Alternatively, `Request` could acccept a parameter to pass on as the `safe` argument to `urllib.quote`.


I'm not against `safe=b"$+!*'(),"`. 


I'd love to be abled to pass my own safe string--I'm running into an issue where '=' is used in the path (not the query string) and requests quoting it is causing the server to 404


A new configuration perhaps?

## 

Kenneth Reitz

On Mar 7, 2012, at 1:41 PM, Mark Thurmanreply@reply.github.com wrote:

> I'd love to be abled to pass my own safe string--I'm running into an issue where '=' is used in the path (not the query string) and requests quoting it is causing the server to 404
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/issues/460#issuecomment-4374303


Sorry for my comment--looks like 0.10.6 works for my case (we were on a pretty old version). A new config may be nice, but it's no longer pressing for my issue.

As a side note, requests is a great library--I'm a huge fan.


Requests v0.10.7 was just released with a new `encode_uri` configuration!
requests should send "Connection: close" if keep-alive is disabled

From Section 14.10 of RFC2616:

'HTTP/1.1 applications that do not support persistent connections MUST include the "close" connection option in every message.'

If keep-alive is disabled, a session object creates a new connection for each request, but each previous connection isn't closed until the connection times out or the garbage collector takes care of it.  This wastes resources on both sides of the connection.

(As an aside, I've noticed that http://httpbin.org/headers returns "Connection: keep-alive" in the headers object even if the client sent "Connection: close").


Keep-alive is almost always on, really. 


@kennethreitz - what if I explicitly want it off?


@yuvadm why?


@kennethreitz for example, if I'm on a server with limited resources and I know for sure that I'm making a single connection that should immediately be closed.


The client will now send `Connection: close`, but I'm not 100% sure the interaction with local connection pooling is correct. In particular, if you set `config={'keep_alive': False}` in the test case at #520, the sockets remain open, but they're in the `CLOSE_WAIT` state instead of `ESTABLISHED`.

I'm going to leave this open until the situation with #520 has been clarified.


I think I understand this a little better now. Probably the Right Thing is for `urllib3.HTTPResponse` to look in the headers and close the connection if the server sent `Connection: close`.
[enh] async.imap added -- generator object handling

standard async map eats huge amount of memory depending on the number/size of requests, imap solves this problem.

sample usage:

``` python
from requests import async

requests = (async.get('http://127.0.0.1/%d' % i) for i in xrange(100000))
for response in async.imap(requests, size=10):
    print response.text
```


Beautiful. Thanks!


\o/
Custom client SSL certificate

Either it's not available in the documentation or not implemented at all, but I'd like to set the certificate I want to use myself.
Ideally that would accept pkcs12, or cert+key PEM file.


You can do a PEM file today. you pass the PEM to the `verify` argument. 

You can also set the `REQUESTS_CA_BUNDLE` OR `CURL_CA_BUNDLE` environment variable.


I just checked this, but it seems the `verify` option is for something a bit different. It allows you to pass your own CA / trust chain and verify the other end against it. What I'm looking for is a way to pass my own key and certificate for the client side. Those parameters are called `key_file` and `cert_file` in urllib3.
I'd like to reopen this bug if possible.


Sorry for the misunderstanding. 

I'd love to support this if the right API can be determined.


``` python
# Default, CA Bundle, like browser. Today.
requests.get(..., verify=True)

# No SSL check. Today.
requests.get(..., verify=False)

# Private PEM
requests.get(..., cert='/path/to.pem')

# Private Cert / Key
requests.get(..., cert=('cert', 'key'))

# No SSL check.
requests.get(..., cert='/path/to.pem', verify=False)
```


I had this same need, and quickly implemented it:

https://github.com/kennethreitz/requests/pull/485

My api might not be as pretty as you'd like, but it seems to work for me.


@chadnickbok this looks great! Moving discussion to that thread.


Moving to #485
Regression: cookie values containing '/' are quoted

This seems to be a similar issue to that described in #440.

This is a regression since Requests v0.7.6 (well, that's the previous version I was using), which did not quote cookie values containing a `/` character. The session cookie that I send to an internal web service stopped working when I upgraded to 0.10.4. The server does not recognize the quoted cookie value sent by 0.10.4. Unfortunately I have no control over the server, so I think changing requests is my only option.


Does the test case for backslashes at #440 suffice to close this also?


I'm not sure. The #450 changeset does not appear to fix this issue. Maybe `/` should be added to the `_RFC2965Forbidden` exception list? I am running a patched version of Requests that does exactly that.

On the other hand, I'm not sure if the conclusion reached in #440 ("that valid cookie chars are any ASCII char excepting comma, colon, backslash, double-quote, control characters and whitespace") is properly implemented by the [current version of oreos/monkeys.py](https://github.com/kennethreitz/requests/blob/ee861f42873ed3b19dd382c3d6fb9e0d5a7521cf/requests/packages/oreos/monkeys.py).


Sorry, that was unclear. The relevant discussion at #440 is not the referenced changeset, but rather the fact that this test case:

https://github.com/kennethreitz/requests/issues/440#issuecomment-5481547

passes under v0.12.0 (master branch / pip). Master is no longer using oreos, it's using `cookielib`.


Ah, I see. Yes, that appears to resolve this issue. I did a quick test here and observe this:

```
$ pip install "requests==0.10.4"
$ python
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print requests.get('http://httpbin.org/headers', cookies={"this_is_a": "slash/"}).text
{
  "headers": {
    "Content-Length": "", 
    "Accept-Encoding": "identity, deflate, compress, gzip", 
    "Connection": "keep-alive", 
    "Accept": "*/*", 
    "User-Agent": "python-requests/0.10.4", 
    "Host": "httpbin.org", 
    "Cookie": "this_is_a=\"slash/\"", 
    "Content-Type": ""
  }
}
>>> 

$ pip install "requests==0.12.0"
$ python
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> print requests.get('http://httpbin.org/headers', cookies={"this_is_a": "slash/"}).text
{
  "headers": {
    "Content-Length": "", 
    "Accept-Encoding": "identity, deflate, compress, gzip", 
    "Connection": "keep-alive", 
    "Accept": "*/*", 
    "User-Agent": "python-requests/0.12.0", 
    "Host": "httpbin.org", 
    "Cookie": "this_is_a=slash/", 
    "Content-Type": ""
  }
}
>>> 
```
'already consumed' accessing response.text with Content-Length: 0

```
>>> import requests
>>> requests.__version__
'0.10.4'
>>> response = requests.get('http://httpbin.org/stream/0')
>>> response.text
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/cduffy/VC/releng-db/virtualenv/local/lib/python2.7/site-packages/requests/models.py", line 774, in text
    content = str(self.content, encoding, errors='replace')
  File "/home/cduffy/VC/releng-db/virtualenv/local/lib/python2.7/site-packages/requests/models.py", line 740, in content
    'The content for this response was already consumed')
RuntimeError: The content for this response was already consumed
```


Does this happen every time for you?


Yes, this is 100% reproducible in my environment.


Funny thing; I have two virtualenvs with requests 0.10.4 and in one of them I get above traceback while in the other one I get `None`.


@piotr-dobrogost, i bet in one you don't have the 'chardet' dependency available.


This is an extremely serious bug. 


https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L742 might be the source: if iter_content() returns nothing - as should be the case with `Content-Length: 0` - it will set _content to None which is also used as a sentinel value.


I'm seeing this error as well, basic traceback:

```
  File "runfats.py", line 117, in runfat
    out.append(ctx.response.content)
  File "/Users/eric/.virtualenvs/fats/lib/python2.6/site-packages/requests/models.py", line 740, in content
    'The content for this response was already consumed')
  RuntimeError: The content for this response was already consumed
```


I'm working on this as we speak.


Fix released.
Warn CookieErrors

#335


Possibly obviated:

```
>>> import requests
>>> c = requests.get('http://www.miniclip.com/games/en/').cookies
>>> c
<<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='ad_is_enabled-position2:homepage', value='true', port=None, port_specified=False, domain='.miniclip.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1336026534, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='ad_is_enabled-showcase:homepage', value='false', port=None, port_specified=False, domain='.miniclip.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1336026534, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='ad_is_enabled-supershowcase:homepage', value='true', port=None, port_specified=False, domain='.miniclip.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1336026534, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='ipLocation', value='US', port=None, port_specified=False, domain='.miniclip.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1367562234, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='language', value='en', port=None, port_specified=False, domain='.miniclip.com', domain_specified=True, domain_initial_dot=True, path='/', path_specified=True, secure=False, expires=1336458234, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False), Cookie(version=0, name='PHPSESSID', value='2332060593395073024-1012474929-3899013470707777536', port=None, port_specified=False, domain='www.miniclip.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=1367562234, discard=False, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>> [(cookie.name, cookie.value) for cookie in c]
[('ad_is_enabled-position2:homepage', 'true'), ('ad_is_enabled-showcase:homepage', 'false'), ('ad_is_enabled-supershowcase:homepage', 'true'), ('ipLocation', 'US'), ('language', 'en'), ('PHPSESSID', '2332060593395073024-1012474929-3899013470707777536')]
```


Good enough for me :)
Recalculating body and content type after hooks

This a patch for issue #GH-445

Perfomance improvements are left to be done. For example, only recalculating when `request.data` or `request.files` have changed.
- File pointers are reset to position 0 to re-read files, when `_encode_body` is called several times.
- _enc_data is recalculated using data which might have changed.


Sorry for the delay on this. Do you think there will be problems with this once we move to streaming uploads?


Integrating the offered solution (42392c9) with streaming uploads will definitely take a little bit of reworking but the time spent reworking things would be worth it.  How soon is the streaming uploads work going to be folded into master?


This is the first time I hear about streaming uploads in requests. But my guess is that as I reset the pointer in the file to 0 using `seek`, this will cause trouble. 

There are many possible alternatives to solve this bug. I've fixed it for the current version of requests. Anyhow, as you understand this is necessary for the hook system to be able to alter the data sent by requests. 

As soon as this solution or any other goes into requests core, I will release a new version of requests-oauth that will be very stable and cover pretty much the whole OAuth spec.

Cheers,
Miguel


I'm more and more tempted to move to only generators on the streaming uploads. Some interesting thoughts on this happened this weekend.


As we've mentioned, boto will need streaming uploads, so we'd love to help however we can here.


Kenneth, 

how long are streaming uploads going to take? I'm thinking you could merge a solution to make the hook system more flexible. Then adapt it when streaming uploads and other features land in. Meanwhile I could unblock the development of requests-oauth.

Also, if you guys don't mind. I think streaming uploads should be discussed in a separate issue. 

Cheers,
Miguel


+1 for getting this merged. See https://github.com/maraujop/requests-oauth/issues/20


This shouldn't be an issue anymore, since we moved the location of the `auth` hook section.
request body expects unicode instead of bytes

The documentation states that :
data – (optional) Dictionary or bytes to send in the body of the Request

However the code in models.py (line 429 in 0.10.4) uses : "isinstance(self.data, str)" to check for non-form data, and "str" was imported from requests.compat as unicode  (str for python 3, unicode for python 2)

This means that:
1. implementation does not conform to documentation
2. unexpected behaviour with python 2.x, for example calling requests.post(url, data="mydata") will NOT result in a non-encoded data, because "mydata" will be a byte string, not a unicode one

I was expecting both unicode and bytes to be considered non-form data parameters


Also the tests hides the problem by importing str from request.compat, which is just wrong, unless importing request.compat is mentioned and required in the documentation. A test script should look and behave just like a regular client of the library.


It seems that this problem is linked to the issue #444. requests.post(url, files=files) fails because the picture can't be converted from byte string to unicode. 


no longer reproducible
Impossible to alter body in hooks

As you can see in this line:
https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L524

Body is precalculated and there is no way to alter it in a hook. It should be calculated after the hook. So if the hook alters `data` or `files`, it gets updated.

Example. If I do this with requests-oauth:

``` python
response = client.post('http://api.twitter.com/oauth/request_token', {'oauth_callback': 'oob'})
```

If in requests-oauth I want to avoid throwing `oauth_*` parameters into body, if header authentication is set. There is no way I can do this. This forces me to add an extra parameter for requests-oauth that is only used in connection initialization.

I'm not submitting a patch, as this can be fixed several ways and it might have some side-effects I'm not aware of.

Cheers,
Miguel


Ok, 

Several of my issues in requests-oauth, needed this to be addressed. I'm submitting a pull request with a fix for this issue, that I've tested against latest requests-oauth `dev` branch https://github.com/maraujop/requests-oauth and it's fully working.

As you understand, what this means, is that people can alter `request.data` or `request.files` in a hook and have that new data encoded and sent.

Please, consider merging this asap, so I can release a new version of requests-oauth with several fixes and upgrades. 

Cheers,
Miguel


Could you please merge this in or fix it your own way, so I can release a stable version of requests-oauth with several bug fixes ?

Thanks


@maraujop

> As you can see in this line:

If you refer to code please remember to link to specific commit not to moving target (head) :)


You are right Piotr, thanks for letting me know :)


I think this should be better now.
UnicodeDecodeError when posting a Multipart-Encoded binary File

I think that there is still a bug related to the bug #403.

``` python
>>> files = {"xml": open("photos.xml", "r"), "image": open("Vienna-Orchestra.jpg", "rb")}
>>> r = requests.post(url, files=files, auth=('***', '***'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "build/bdist.linux-i686/egg/requests/api.py", line 84, in post
  File "build/bdist.linux-i686/egg/requests/api.py", line 39, in request
  File "build/bdist.linux-i686/egg/requests/sessions.py", line 200, in request
  File "build/bdist.linux-i686/egg/requests/models.py", line 535, in send
  File "build/bdist.linux-i686/egg/requests/packages/urllib3/connectionpool.py", line 370, in urlopen
  File "build/bdist.linux-i686/egg/requests/packages/urllib3/connectionpool.py", line 246, in _make_request
  File "/usr/local/lib/python2.7/httplib.py", line 946, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python2.7/httplib.py", line 987, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python2.7/httplib.py", line 940, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python2.7/httplib.py", line 801, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 484: ordinal not in range(128)
```

I've made different tests and always get the same problem. It works quite well if the photo is a simple text file.


I'm trying to solve the problem in urllib3/connectionpool.py line 370 by this way:

``` python
# Make the request on the httplib connection object
            httplib_response = self._make_request(conn, method, url,
                                                  timeout=timeout,
                                                  body=unicode.encode(unicode(body, errors='ignore')), headers=headers)
```

The photo and XML file are now sent to the server without errors. But the photograph can't be displayed since it contains less information than the original (due to "errors='ignore'")...


It looks to me like this should be filed against shazow/urllib3


If anyone runs across this issue, be sure that all the arguments you pass into requests are byte strings (besides just the payload or `data`). Any unicode instances making their way in will pollute the message building in httplib (in my case it was passing in a unicode u"POST" method).


So I don't understand why I'm still getting this error? I'm not passing any unicode. Trying to post a binary file via:
<code>response = requests.post(url, files=files, auth=oauth, params=params, verify=False)</code>
Tests cannot be run successfully after cloning the repo

There seems to be some fiddling to be done before the tests can be run successfully right after a clone. Changes in this pull request is how I got them to work on my clone.


Okay, this pull request looks slightly messed up.

Kenneth, I can create pull request with just the changes of adding dependencies to requirements.txt and removing the unused envoy import. And close this one. What say?


All of these changes are unnecesary


> But, what I'm saying is one should be able to just clone, get deps, run tests, modify code, run tests.

True. Dependencies needed to run tests should be well documented and there should be some easy way of installing them. Btw, nose is in no way a requirement and should not be listed as such.

The easiest way of making some progress in this matter is to create requirements-tests.txt file with a list of dependencies and information when each dependency is needed.


> nose is in no way a requirement and should not be listed as such

Yep. Nose being listed there gives a (false?) impression that nose is all that's necessary to run the tests successfully. :)

> The easiest way of making some progress in this matter is to create requirements-tests.txt file with a list of dependencies and information when each dependency is needed.

I like this idea. Possible to get this accepted?


test_requires in setup.py is a nice place for testing requirements
There is also extras_require for extra stuff (optional feature in module)
Or setup_requires for setup time dependency

There is very little documentation about those options.


I don't think it's really a big deal, honestly.


Nope indeed, but that's the pythonic way to do it.
Should requests honor http-equiv meta refresh?

It's possible for a server to instruct a client to redirect to a URL by using the [http-equiv meta tag](http://en.wikipedia.org/wiki/Meta_refresh).  It's antiquated, but it's still out there, and browsers still handle it as a redirect.

Should requests honor the http-equiv refresh, handling it essentially the same as a 301?  I'd like to think so, because that's what browsers do.  To do so would mean having requests parse the <head> section of the response body -- which shouldn't be too difficult or inefficient (even without BeautifulSoup ;).  Has this been considered/addressed?


IMO this seems very much overkill for an HTTP library—not all HTTP is HTML, and even when it is, this would be something that could be pretty easily built by a user who needed it.


Like @umbrae said, this is an HTTP library, not an HTML library.


Fair points, I definitely respect the desire to keep the library small and focused, and love requests for what it is.  For me the question came down to how much you want requests to be able to quack like a browser.  I guess the more fitting solution for that need is mechanize.

[Here's a thread](http://www.gossamer-threads.com/lists/python/bugs/459451) from way back when John Lee added support for http-equiv into "ClientCookie" (which later became mechanize), including a debate about whether/how that support should be integrated into urllib2.  It wasn't, but mechanize's HTTPEquivProcessor is available for use with urllib2 to do what I want -- hande http-equiv tags as if they were HTTP headers.
Make Session objects picklable

Currently, when we try to pickle a `Session` object, we get this:

```
  File "[...]python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle lock objects
```

The `Session` objects can be made pickleable rather easily. If you are willing to accept this functionality, I can create a pull request. I showed the [implementation](http://sharats.me/serializing-python-requests-session-objects-for-fun-and-profit.html#implement) on my blog post about this.


Broken link


Sorry, fixed now.


Let's do it!


I committed my changes and referred to this issue number in the commit message. Can you pull with that or should I create a new pull request (which creates a new issue I believe)?

Edit: Apparently there needs to be a pull request created.


Awesome!
SPNEGO auth failed on master vs 0.4.1

hi ,
first thanks for this awsome lib.
my code is working successfully under 0.4.1.

moving to master or 0.10.2 i noticed the following :
   passing a cookiejar object does not work but a dict is nicer anyway and access cookies directly from response is nice.

but some of the requests i m doing with master keep on getting a SPNEGO authentifcation failed.

any idea? would really like to move to the latest requests.

Thanks 


I just tested different version and the latest working version is 0.7.6 where i don't get the SPNEGO authentication. i can at least use my refactored code from 0.4.1 to 0.10.2 ( using response.cookies...) with this version and hope for a fix.
just tested 0.10.3 with the same problem.

I suppose the problem is due to cookie changes and redirect changes introduced in 0.8.0. 

looking at the different commits between 0.7.6 and 0.8.0 : checking out f5bbb9714d66103280d0ca10742877f4522787ff works and checking out 498b446f141675026ae7364b922f03cde1de728c introduces the SPNEGO error. quite difficult to find the part which breaks thing as merge's have not been squashed


Cookie handling has been refactored. Try revisiting the issue now? Also, is it feasible to provide a test case?
Requests gets 404 but it shouldn't be

When accessing a simple website, Requests get 404 error while Restkit success with a 200 :

``` python
>>> l = requests.get("http://www.ecrans.fr/OVH-L-Internet-giga-familial,14088.html")
>>> l.status_code
404

>>> r = restkit.request("http://www.ecrans.fr/OVH-L-Internet-giga-familial,14088.html")
>>> r.status_int
200
```


Are you using 0.10.2?


Nop, I used 0.10.1. I tried with 0.10.2 and it works fine. Thanks !


There were some improvements to the escape mechanism.


This improvements closed the issue #404 too
ValueError in generate_chunked (invalid literal for int())

Looks like there's some trouble handling some specific URLs in the version 0.10.2 of requests:

```
>>> import requests
>>> requests.head('http://feeds.gawker.com/~r/jalopnik/full/~3/gaYrXc40sbA/this-is-the-worst-school-zone-sign-in-history', allow_redirects=True)

[...wait a very long time...]

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/api.py", line 73, in head
    return request('head', url, **kwargs)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/api.py", line 39, in request
    return s.request(method=method, url=url, **kwargs)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/sessions.py", line 200, in request
    r.send(prefetch=prefetch)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 557, in send
    self._build_response(r)
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 211, in _build_response
    r.content  # Consume socket so it can be released
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 746, in content
    self._content = bytes().join(self.iter_content()) or None
  File "/Users/umbrae/Envs/rdb/lib/python2.6/site-packages/requests/models.py", line 676, in generate_chunked
    pending_bytes = int(pending_bytes, 16)
ValueError: invalid literal for int() with base 16: ''
```

I can scrape up a few more test case URLs as well if you'd like.


Looks like you may need to now pass `allow_redirects=True` to trigger this issue, still exists from what I can tell though.


redirection will follow redirects, while doing so it tries to consume the content
which fails for chunked encoding on head (no content)

it seems there is need for more conditions on when to have custom chunked encoding parsing
(the custom chunked reading is necessary since httplib is too incappable/broken to correctly pass over chunks/lines)

a solution would be to check for Empty Data and break out of the loop (as httplib handling incorrectly would)


@RonnyPfannschmidt Thanks for the insight Ronny.


@RonnyPfannschmidt care to add it? :)


no


Simplified chunks are back now. This shouldn't be a problem anymore.


:+1:
http post's request.content works in 0.9.1 and is None in 0.10.2

I have a project using requests for a while on version 0.9.1. After pip installing the new 0.10.2 version in my virtualenv,  a post request using HTTP digest auth is failng with request.content == None. 

```
def create(self, **kwargs):
    r = request.post(uri,
                           auth=HTTPDigestAuth(username, password),
                           headers={'host': host,
                                    'content-type': 'application/json'
                                },
                           data=json.dumps(kwargs))
```

I believe this is a regression as simply pip installing 0.9.1 causes the code to work unchanged.


Seems to happen only if the response is empty:

```
echo -e  "HTTP/1.0 200 OK\nContent-Length: 0\n\n"|nc -l -p 8080 &
python  -c 'import requests; r = requests.post("http://localhost:8080/"); print repr(r.content)'
None

echo -e  "HTTP/1.0 200 OK\nContent-Length: 1\n\na"|nc -l -p 8080 &
python  -c 'import requests; r = requests.post("http://localhost:8080/"); print repr(r.content)'
'a'

```
Use correct path when opening file in tests

When running `make test`, several tests failed with a similar error:

``` bash
$ make test
nosetests --with-color ./tests/*
............EEE..............................................EEE...................................................EEE..................................
======================================================================
ERROR: test_POSTBIN_GET_POST_FILES (test_requests_async.RequestsTestSuite)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/home/christopher/code/requests/tests/test_requests.py", line 294, in test_POSTBIN_GET_POST_FILES
    with open('test_requests.py') as f:
IOError: [Errno 2] No such file or directory: 'test_requests.py'

(...)

----------------------------------------------------------------------
Ran 152 tests in 11.122 seconds
FAILED (errors=9)
```

The problem was that the current working directory was the root of the project, so calling `open('test_requests.py')` failed because it's now in the tests directory.

After the fix:

``` bash
$ make test
nosetests --with-color ./tests/*
................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 304 tests in 19.953 seconds
OK
```


Using `__file__` now.


Cool, that makes more sense.
URI cleanup of non-path components

My [previous branch](https://github.com/mgiuca/requests/tree/py3-uri-encoding) fixed URI cleanup for path components. This branch extends that fix to address Issue #429, which requires URI cleanup for the query part of the URI.

I have added comprehensive tests for the sanitisation of the query part of the URI, and rearranged the call to `utils.requote_path` so that it is now applied to the whole URI after it is constructed, and not just the path. Note that this required that all URI components be converted to UTF-8 before the URI is reconstructed (since the URI is now sanitised _after_ reconstruction). I have renamed `utils.requote_path` to `requote_uri` to reflect its new role.


This is fantastic. Thanks a ton. I really appreciate this.


:cake:
Enhance max_retries functionality

Right now, the max_retries config looks like it just handles bad connections in the connection pool down in urllib3. What would you think about adding some higher level retry logic to Requests? For example, if a request ended up with a 503 it might be nice to have some retry logic built into requests to try to re-submit the same request again. The number of retries would be controlled by a new arg with a default of 0.


I think retries should, in general, not be used. I'd rather provide an easy mechanism to send a request again. Currently, you can do this by calling `Request.send(anyway=True)`
Installation of v0.10.1 on Windows 7 & Py2.5 fails - syntax error in utils.randombytes

ver 0.10.1 contains new implementation of `requests.utils.randombytes`.
line https://github.com/kennethreitz/requests/blob/v0.10.1/requests/utils.py#L185
causes an error on Windows 7 with Python 2.5.4

Older versions 0.10.0 & 0.9.x works.


Python 2.5 is no longer supported.


Why is 2.5 no longer supported?


I'd say because it's old, and prevents from using new stuff of 2.6+ versions.


Yep, but still there's a lot of code that runs on Py2.5. Sad but true. I guess I'll need to use older version to run it on Py2.5. Should I close an issue?


@kennethreitz Hm, I thought you would consider adding Python 2.5 support again.. at least that's what I understood when I asked you  about it regarding requests adoption in pip. Mind elaborating?


I have no intentions of supporting Python 2.5 at this time. If pip decided to move to requests, I would revisit that ;)


@kennethreitz Not sure what more is needed than a pip developer saying that requests adoption requires 2.5. A public statement?


Start adopting it and I'll start putting 2.5 back in. It'll take half an evening :)


Work on the adoption is being done as we write these silly messages, I've ported most of the PEP381 changes to requests. I need to test it on Python 2.5 though :)


Fantastic :)  I'll start to add 2.5 back in soon then. 


I lied.


Obviously.


:cake:


Should we move this discussion to a new issue? The name of this issue is totally misleading...


@Lukasa Good idea +1