Raibek/requests-github-issues huggingface dataset:
Partial content and compression : decoding error

Be default requests generate http requests with 'Accept-Encoding: gzip, deflate' header.
If one want to GET only the last 5 bytes of a file, adding a Range: -5 header should do the trick. However, remote server may honor the Accept-Encoding and return the last 5 bytes of gzipped content (instead of the gzip of last 5 bytes as one may think).
This is the difference between Accept-Encoding and transfer encoding 'TE' headers. 'Accept-Encoding: gzip' is for fetching the compressed entity, TE: gzip is for indicating to next proxy that we support a response with header 'Transfer-Encoding: gzip'. In both cases response body is compressed data ; however if Range header specifies partial content, the difference is _when_ compression is achieved :
- before slicing for Accept-Encoding
- after slicing for TE
  [however this is theoric, and not really supported by web servers and proxys]

In my case, the bytes received can not be decoded and urllib3 raises with error :
File ".../site-packages/requests/models.py", line 678, in generate
    raise ContentDecodingError(e)
requests.exceptions.ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))

Error is probably in urllib3.

Some requests questions :
Is it possible to disable compression ?
Is it possible to get the raw gzipped response (without other decoding than chunked) ?


> Is it possible to disable compression ?

Yes. Set `headers={'Accept-Encoding': None}`.

> Is it possible to get the raw gzipped response (without other decoding than chunked) ?

Yes. If you set `stream=True`, you can then call `response.raw.read(decode_content=False)`


In the future, please ask questions on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).
Handle empty chunks

Empty chunk in request body could prematurely signal end of chunked
transmission. As a result, the terminating zero-size chunk sent by
'requests' can be interpretted as bad request by the recepient. I
have used the same logic used by httplib to handle such cases.


Resubmitted https://github.com/kennethreitz/requests/pull/2388 on proposed/3.0.0 branch.


:+1: LGTM


LGTM as well. =) We should note this change somewhere before we merge it and lose it.


Changes are noted in https://github.com/kennethreitz/requests/pull/2646. Merging. :sparkles: :cake: :sparkles: 


Thanks @neosab
Connection pool not recycling connections on timeout or connection exceptions

If preload_content is not specified on a request, connections are implicitly returned to the pool after the request is read. However, when a timeout error occurs or the connection is dropped the connection is closed but not returned in to the pool (since the response is None). This problem is compounded when retries are turned on since the next attempt will grab a new connection from the pool thus depleting it further. With non-blocking pools this is not a problem since we will create a new connection if the pool is empty but when the pool is blocking we have found that eventually the pool size becomes zero (after no.of connections timeout errors) which causes the calling application to hang on it's next request.
This can be fixed in connectionpool.urlopen in the exception handlers by explicitly returning the closed connection to the pool via _put_conn (if release_conn is false), since subsequent calls to _get_conn will check if the connection has been dropped and return a new one.


This is a bug in [urllib3](https://github.com/shazow/urllib3): can you please open it there?


Thanks for the response. Yes I will do that.
expose request start time

When implementing time based throttle mechanism, the start of request time is quite useful, please consider expose it as instance property, so we don't need to wrap requests in a multiple `time.utctime()` calls. It's a waste of syscall and duplicated code.

https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L570


I don't think we're going to expose it. We weren't keen to expose the `elapsed` attribute in the first place. We're not going to add yet another attribute, especially one we've never had a request for before. `elapsed` was only necessary because the hooks provided by requests were severely trimmed in 1.0.


If no one comments in the next 6 days, I'm going to close this.


I am ok with closing this,  as long as `elapsed` will be obsolete


## Why do you want a widely used feature to be removed? 

Sent from my Android device with K-9 Mail. Please excuse my brevity.


@sigmavirus24 Pardon my poor English, but how could  `elapsed` be widely used and you weren't keen to expose it?

My rationale is like this, since `elapsed` is widely used, and it involves a system call to get start request time, so it'd better to directly expose the start request time as a property directly, so other developers can make more use of this measure without wrapping yet another get current time system calls. As you can see people use `requests` often involve with time based constraints and there would be a ton duplicated code.


> how could elapsed be widely used and you weren't keen to expose it?

It wasn't widely used before we added it. We were hesitant to add the attribute in the first place but (like most things that are added to public APIs) people use it because it's there. It's widely used now and it's not something we're going to remove.

There's two things to consider though:
1. Requests is in a feature freeze at the moment. That means no new additions to the API.
2. You are the first person in several years to ask us to expose this feature. Without a large number of people needing this to be provided by requests, the default answer is "No."

That said, you keep complaining about needing this causing you to make another system call. If you think system calls are inhibiting the performance of your application, I would strongly urge you to do real benchmarking to determine what is the actual cause of performance degradation. If you've already optimized every part of your application and have found that these system calls are causing performance problems that are noticeable, I think you have other problems to solve.


Now I feel stupid since me alone is complaining this.

This must be a bad idea. Sorry for this, I'll see myself out.
Use keep-alive / streaming while processing

Is there a way to send intermittent responses to client to avoid timeout for long-running tasks (~2minutes) ? 

If not, I guess I will have to do the following flow: 
1. User requests a webpage
2. Check if that webpage is available in cache
3. If not, redirect them to a holding page that checks ~15 seconds if that request is ready
4. Load the response

Seems less than ideal, but gets the added benefit of showing some sort of progress indicator to the user.


@kvarga sorry, are you using the requests library? You seem to not be using it at all unless I'm misunderstanding your bug.


Oops. I meant to post this to flask. 


No worries! :8ball: says the flask maintainers will be more helpful.
why implement requests.util.timeout.Timeout() instead of bsd socket options?

In python's socket module, `socket.timeout` could be used as connect timeout, also `setsockopt` can be used with `SO_RCVTIMEO` and `SO_SNDTIMEO`. It there a reason to implement our own  `requests.util.timeout.Timeout` class instead?


Are you looking at the right project? https://github.com/kennethreitz/requests/blob/master/requests/utils.py


More generally, the answer is yes. `Timeout` is not special: it's just a wrapper over the timeout functions available in the socket module. The reason it was written into urllib3 (which is where I think you meant to say) is that it is useful to have an abstraction that is at the same layer as urllib3. Generally, passing things from the `socket` module into `urllib3` should be considered an expert-level action and only taken rarely: timing out requests is good practice, and should be done often.


Ah yes, I meant this file

https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/util/timeout.py

Thanks for the explaination!
Connect Error or Timeout Error

At https://github.com/kennethreitz/requests/blob/8b5e457b756b2ab4c02473f7a42c2e0201ecc7e9/requests/models.py#L664 the urllib3 `ReadTimeoutError` gets mapped to a `requests.exceptions.ConnectionError`. It seems like the `requests.exceptions.ReadTimeout` would be a more obvious exception.

Is this just an oversight, or is there a reason for keeping it as `ConnectionError`?


Good spot! I think this is an oversight.

Unfortunately, changing the exception type is a backward incompatible change, because `ReadTimeoutError` does not inherit from `ConnectionError`. I think we should do it, but we can't do it until 3.0.0.


Sorry, just noticed this is a duplicate of https://github.com/kennethreitz/requests/issues/2392


Ah thanks @jongiddy. It's been long enough (over a year, wow) that I completely forgot that was open. Would you mind if we closed this since it's a duplicate?


No problem. I would have closed it myself, but wasn't sure how that affects the labels and milestone info.


It doesn't affect it. You're still welcome to do so yourself. I'll make a reminder to close it next week if it's still open


Closing as duplicate.
get url redirect hangs in requests

I am trying to handle url redirect using requests,but it hangs.The original python lib works as expected.Below is the code(**Python 3**).

``` python
import urllib.request
import requests

url = "http://go.microsoft.com/fwlink/p/?LinkId=522148"
final_url = urllib.request.urlopen(url)
print(final_url.geturl())

r = requests.get(url)
print(r.url)
```


The code below also works fine:

``` python
r = requests.head('http://github.com', allow_redirects=True)
print(r.url)
```

should I use **requests.head** over **requests.get** when handling url redirect?


@gengjiawen This URL redirects to a 4GB big ISO image. So the download takes a little bit.
You probably don't want to read this into memory anyways.
Try `requests.get(url, stream=True)` and then iterate over the content.


@t-8ch Thanks, your solution also works.
Move native string conversion to Request

This fixes #2613. I believe that this technically constitutes a backward-incompatible API change, so I've proposed this against 3.0.0. Let me know if you disagree @sigmavirus24 (the other option is really that this is a bugfix), and I'll propose against master.


hehehe The tests are trying to pass `None` as a method. We should either change that default or fix the tests.


Grr.


Stupid question: Why default to an empty string if method is None? I'm not entirely comfortable with that


Yeah, it was definitely a bit of a blind shot. A better idea might just be to have no default for `method`. I can't think of anything else non-surprising.


@Lukasa thoughts on the last commit that I pushed?


Note that if we want to present a better error to the user, we can catch the error raised by `to_native_string` and raise a new one.


If we're going this route, we should remove the default argument from `method`, as it cannot safely be used any longer.


I feel like using an empty string as a default value is a footgun. That said, I don't feel strongly about whether method should be a required argument. Currently, in our own code we do, one might do

``` py
req = requests.Request()
self.update_request(req)
prepped = req.prepare()
```

I think this is something valid for someone to do. That said, perhaps `req.prepare` could raise a `ValueError`, `TypeError`, or something more descriptive that we include in `requests.exceptions`? That stops the user from shooting themselves in the foot and then we just move the `to_native_string` into `Request.prepare` or into `PreparedRequest.prepare_method`. In fact, I think that `PreparedRequest.prepare_method` should actually be where we raise the exception. Currently, we do nothing if `method` is `None` and I'm pretty sure that's bad.


I can see the advantage in that approach, I'd be ok with that.


@Lukasa updated with that approach.


ping
fix unittest typo

IDE auto-complete error.  

edit:  edited to be a bit more pythonic


@colindickson Thanks for this! Rather than merge two identically named commits and a merge commit I rebased them down into fd31453aa25f80f04e1ce36de9abff0460bd136a, and merged it in 8b5e457b756b2ab4c02473f7a42c2e0201ecc7e9.

Thanks again! :cake: :sparkles:
[Bug?]  json.dumps(json)?

download from https://pypi.python.org/pypi/requests/2.7.0
models.py
line 418
bug?


Do you mean https://github.com/kennethreitz/requests/blob/b7bd29734022e634634d908e10bbf7a5ac15e76b/requests/models.py#L428 ? If so, no.


Yes,“json_dump” is True
But in  kennethreitz-requests-v2.7.0-8-g4505f71.zip   there is   “json.dumps”
Fixed right now
Tks :) 


There is no such file on PyPI. If you're downloading files from GitHub, I'd advise you not to.
fix contextlib.closing bug for sessions where content is not consumed…

… (issue #2593)


Why are you making 25 requests to three websites that we don't use anywhere else in the test suite?


mainly as an attempt to reproduce the bug demonstrated in the original issue.   i suppose i could simplify it a great deal


Please don't create a new pull request. If you continue to push commits to this branch they'll just show up here.


Also, let me be clear, your fix here is 100% correct.


test should be better now :) 


LGTM. :cake:
UnicodeDecodeError when post a local image file

i am using python 3.4, and request 2.7
i am trying to post a local jpg image like this, and got encoding error:

```
    import requests
    response = requests.post("url here",
      params={
        "image": open("path to local image", mode="r")
      }
    )
```

if i change the mode to 'rb' the error is gone:

```
    import requests
    response = requests.post("url here",
      params={
        "image": open("path to local image", mode="rb")
      }
    )
```

the error details are listed below:
'Traceback (most recent call last):
  File "test_facemark.py", line 7, in <module>
    "image": open("path to the jpg file", mode="r")
  File "c:\python34\lib\site-packages\requests\api.py", line 109, in post
    return request('post', url, data=data, json=json, *_kwargs)
  File "c:\python34\lib\site-packages\requests\api.py", line 50, in reques    response = session.request(method=method, url=url, *_kwargs)
  File "c:\python34\lib\site-packages\requests\sessions.py", line 451, in
t
    prep = self.prepare_request(req)
  File "c:\python34\lib\site-packages\requests\sessions.py", line 382, in
e_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "c:\python34\lib\site-packages\requests\models.py", line 304, in pr    self.prepare_url(url, params)
  File "c:\python34\lib\site-packages\requests\models.py", line 397, in prurl
    enc_params = self._encode_params(params)
  File "c:\python34\lib\site-packages\requests\models.py", line 92, in _enarams
    for v in vs:
UnicodeDecodeError: 'gbk' codec can't decode byte 0xff in position 0: illeltibyte sequence'


In Python 3, if you open a file in `'r'` mode, that will open the file in _text_ mode. This means that Python will attempt to use your system encoding to decode the bytes in the file. That is causing this exception here.

When uploading binary data to the web, you should always open the file in bytes mode: `'rb'`. You _should_ also do that with text files, really, to avoid problems with an encoding round trip.
Don’t break IPv6 address format

No idea what this does to older supported Python versions. In Python 3.4.2, at least, you can’t strip away the brackets otherwise no IPv6 address will resolve.


Thanks for this!

Two notes. First, this is a change in [urllib3](https://github.com/shazow/urllib3), please open this pull request on that repository. Secondly, given the comment above I don't think we can unilaterally remove this logic. Likely we'll have to detect what versions are affected.
SSL Error 14077438

Certain cloudflare websites cannot be access with requests. I have dis problem across different distros. Same problem here: https://github.com/SiCKRAGETV/sickrage-issues/issues/1231

requests.get('https://sceneaccess.eu')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 69, in get
    return request('get', url, params=params, *_kwargs)
  File "requests/api.py", line 50, in request
    response = session.request(method=method, url=url, *_kwargs)
  File "requests/sessions.py", line 465, in request
    resp = self.send(prep, *_send_kwargs)
  File "requests/sessions.py", line 573, in send
    r = adapter.send(request, *_kwargs)
  File "requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error


Where have you installed requests from, and what versions do you have installed?


This is likely because you're linked against an old version of OpenSSL. Can you run this for me and show me the output?

``` bash
python -c 'from cryptography.hazmat.backends.openssl.backend import backend; print backend.openssl_version_text()'
```


Yup, I can confirm: if I have PyOpenSSL compiled against OpenSSL 0.9.8 the handshake fails because there are no overlapping cipher suites. If I compile against OpenSSL 1.0.2, however, the handshake succeeds.


@Lukasa it's probably best to use print() for instructions like that in case the end user is on a distro which has changed the sym link for python to point to python 3 instead of 2.

As a side note, I generally recompile python after installing a new version of OpenSSL, so I was a little surprised to see my workstation here reporting the version as 1.0.2 instead of 1.0.2a and it turns out that the cryptography module requires a force reinstall to detect that change in addition to whatever is done to the python installation (which normally includes byte compiling installed modules), it would be quite easy for anyone to overlook that even after upgrading OpenSSL.


> it's probably best to use print() for instructions like that in case the end user is on a distro which has changed the sym link for python to point to python 3 instead of 2.

That's not really relevant. =) The number of users using Python 3 instead of Python 2 is very low, and I trust that most users capable of reporting bugs on this bug tracker are also capable of translating this extremely common line of code into the correct syntax for their platform. Given that `print()` won't work on Python 2 without an appropriate **future** import, I'm choosing to optimise for the more likely case.

Regardless, I think we can close this. @sergedroz, your OpenSSL is too old. If you're using pyOpenSSL, please reinstall cryptography linked against a more modern OpenSSL. If you're not, please upgrade to a newer version of OpenSSL and then rebuild Python.
Auth handler should be calling close, not release_conn

On [this line](https://github.com/kennethreitz/requests/blob/956ec8321a76f0b5a9f5bcb591cf6e77d773f819/requests/auth.py#L182) we should be using the response's "close" method. We shouldn't be explicitly calling a method on the "raw" attribute.


Resolved in #2601.


Should I add a test to ensure close gets called?
Would using a mock object be the right way?
Partially revert ab84f9be5740d4649d734e73b84f17f85e52ffc9

As discussed in the comments for ab84f9be5740d4649d734e73b84f17f85e52ffc9, this code block is necessary thanks to our shiny `json` parameter.

As a temporary workaround, this reinstates the code Kenneth hates so much so that the builds start working again. @kennethreitz feel free to replace this with something else longer term if you still hate it. :grin:


`import json as notsimplejson` :P


`import json as complexjson`? :grin:


`import json as _json` Really anything to make sure this works properly is a good thing.


ping @sigmavirus24 


:+1: but the tests still seem to be broken because other things were relying on `json` being a module defined in `requests/models.py` http://ci.kennethreitz.org/job/requests-pr/662/PYTHON=2.7/console


I'm doubling down on complexjson here.


:cake: 


Forgot something...

:sparkles: :cake: :sparkles:
colon in get params is double encoded

Thanks for maintaining requests! I love it!

If a parameters in <code>params</code> passed to <code>get()</code> includes a colon (as is required for categories in the MediaWiki AP, for exampleI) requests double encodes it. Here's some example code:

```
import requests    
parameters = {'action' : 'query',
              'list' : 'categorymembers',
              'format' : 'json',
              'cmtitle' : 'Category:London'}
r = requests.get('http://wikivoyage.org/w/api.php', params=parameters)
print(r.url)
```

This returns a double escaped colon as <code>http://en.wikivoyage.org/w/api.php?action=query&cmtitle=Category%253ALondon&list=categorymembers&format=json</code>.

A working URL would be: <code>http://en.wikivoyage.org/w/api.php?action=query&cmtitle=Category%3ALondon&list=categorymembers&format=json</code> or even the unescaped <code>http://en.wikivoyage.org/w/api.php?action=query&cmtitle=Category:London&list=categorymembers&format=json</code>, both of which work fine.

Shouldn't a colon in a parameter only be escaped once as "<code>%3A</code>"? Composing the URL and passing it directly in seems to work just fine.


Requests does not double encode the colon. =)

What you don't see in that code is that you've been redirected. That URL is actually where you ended up after a chain of redirects. If you look for the _first_ request in the chain, you'll see the correct URL:

``` python
>>> r.history[0].url
u'http://wikivoyage.org/w/api.php?action=query&cmtitle=Category%3ALondon&list=categorymembers&format=json'
```


Got it! Thanks for the incredibly quick response and resolution! The URL seems to also be correct in the second step in the chain.

Is this a bug in the redirect code that WikiVoyage is using or is this an unavoidable side effect of relying on the 301s? This is simple enough to fix on my end but I'm wondering if I should file a ticket with the Wikimedia Foundation.


I think the bug is with WikiVoyage, it seems to be incorrectly parsing the URLs. The bug is not in the first redirect (from wikivoyage.org to www.wikivoyage.org), but in the second redirect (from www.wikivoyage.org to en.wikivoyage.org). You can therefore probably avoid the bug in your own code by going straight there, and then file a bug with whomever is managing that code to check why it mishandles URL escaping.


Thanks! I had already fixed the bug by avoiding the redirects but I wanted to let Wikimedia (who run WikiVoyage) about the issue so that others who make the same mistake don't run into the issue.

Thanks again for your help!
Redundancy when requesting URLs

This is just a question I'd like to mark as feature request.

Is it ever planned to provide some kind of redundancy - i.e. add a list or URLs to the Request Object and the Request Object would try to fetch the next URL, if the first one timed out.
This might be interesting if requesting web APIs, where redundancy is not provided by the service.

Just tell if this is a bad idea.


Thanks for the feature request @cornelinux!

This seems like the kind of thing that could easily be built on top of requests. =) If a `Session` object is used, it shouldn't be hard to write a little wrapper around it that does exactly that.

However, I don't think we'll be adding this feature to requests core, as we're in an indefinitely feature freeze. Sorry!
Session post() sending a GET instead of a POST intermittently

I can't figure out how this would be happening by looking into the source code but here is the behavior I am seeing:

```
>>> import requests
>>> s = requests.Session()
>>> r = s.post('http://mydomain.com/login/attempt', data={'username': 'user', 'password': 'pass'})
>>> r.request
<PreparedRequest [POST]>
>>> r2 = s.post('http://mydomain.com/api/v1/something', data={'name': 'foobar'})
>>> r2.request
<PreparedRequest [GET]>
>>> r3 = s.post('http://mydomain.com/api/v1/something', data={'name': 'foobar'})
>>> r3.request
<PreparedRequest [POST]>
```

I'm using Requests 2.7.0. Let me know if I should provide any other debug information.


`Session.post` is not sending a GET, you're being redirected. Check `r.history` for each one.


Thank you! user error.
Can't post BytesIO file-like object

Hi. I'm trying to zip a directory in memory and then POST that to a server. Unfortunately, I don't seem to be able to use the BytesIO object to which the zip data is written. Instead, I have to get the bytes from it. But the docs seem to indicate I can provide either bytes or a file-like object in the tuple.

Here's a condensed version of the code:

```
zipData = io.BytesIO();
with zipfile.ZipFile(zipData, mode="w") as zf:
    zf.write(<some content>)

url = "https://some.host/api/upload"

files = { "ipa" : open(ipaPath, "rb") }
#files["dsym"] = (zipName, zipData)             # file-like object doesn't work?
files["dsym"] = (zipName, zipData.getvalue())   # works
r = requests.post(url, files=files)
```

It may well be user error; if so, sorry for the noise.


So try this on for size:

``` py
zipData = io.BytesIO();
with zipfile.ZipFile(zipData, mode="w") as zf:
    zf.write(<some content>)

zipData.seek(0, 0)
url = "https://some.host/api/upload"

files = { "ipa" : open(ipaPath, "rb") }
#files["dsym"] = (zipName, zipData)             # file-like object doesn't work?
files["dsym"] = (zipName, zipData.getvalue())   # works
r = requests.post(url, files=files)
```

The problem is that the `BytesIO` object moves the position when you write to it. If you do

``` py
fp = io.BytesIO()
fp.write(b'some data')
assert fp.read() == b''
fp.seek(0, 0)
assert fp.read() == fp.getvalue()
```

Cheers!


Yep! That was it! Totally explains what I was seeing, too (empty content). Thank you.


Happy to help!


Perfect Answer. Always worked with files when we open default point is at start in case of write to streams we have to explicitly put the pointer back at start of the object. Saved my day thanks for the explanation.
Remove the __bool__ and __nonzero__ response methods

Many people expect to be able to say:

```
response = make_request(url)

if response:
    body = response.content
```

Where the first part should test for whether or not response is None.
Instead, the **bool** and **nonzero** methods return response.ok, so if
the response is actually a 4xx or 5xx response, then the user would
expect to get the body of the response.

By removing these methods, we restore the functionality that most users
expect.

Closes #2002 

---

**Note** this is development for 3.0.0


For the 3.0.0 stuff, I think we should try to be adding changelog updates for this: ideally, fairly thorough ones.


Otherwise, this looks totally reasonable. =)


Since we've discussed keeping this branch around for a while and the likelihood of having another release (at least), I made a different file to store the 3.0 history to avoid messy merge conflicts. Let me know if you just want it in `HISTORY.rst` instead


I think you accidentally a word, but this is otherwise perfect.


The word is unaccidentallyied
Cut 2.7.0 this weekend

Ok, I believe we've left enough time for the chunked stuff to bake, and no further issues seem to have come out of the woodwork. At this point, I think we should cut 2.7.0 and begin the switchover to our new release process.

@shazow are you open to tagging a new urllib3 release, and if so what commit would you like us to use? If you don't really care, by default we'd use shazow/urllib3@74073791a3429d9b9f375563954f57f2181599dc.

Things to do:
- [x] Update urllib3.
- [x] Confirm that the #2455 problem sites still work.
- [x] Update changelog.
- [ ] Tag.
- [ ] Push release.
- [ ] Have a drink.


In my ideal world we'd update the `certifi` bundle as well, but I'm now strongly averse to trying to do too much with these releases. @sigmavirus24?


I'm not strongly opposed to it. I'd like confirmation that the problem sites in #2455 at least work. Thoughts?


Agreed. Adding it to the list.


Alright, the chunked keep-alive problem does appear to be resolved after synchronizing urllib3.


Hm, this downgraded the embedded version string of urllib3.
I think it is confusing to have a quite arbitrary version in the bundled library.

Thoughts:
- Wait for a @shazow to cut a new release and update the version in urllib3 itself, then pull that.
- Rewrite the version string to be actual the commit id.


Tagged and pushed urllib3 v1.10.4.


And I updated the commit to resolve that problem. The only delta between the previous commit and the new one is the version identifier.


@sigmavirus24 Any objections to me merging this PR (manually without a merge commit, because we don't need it) and pushing the release?


0 (zero) objections here.


Done.
ImportError: No module named 'requests.packages.chardet.sys' when frozen by PyInstaller

When running a simple test-program frozen by [PyInstaller](http://www.pyinstaller.org) this program fails with an `ImportError: No module named 'requests.packages.chardet.sys'`. The same program runs fine when run unfrozen (which means: directly from the command line).

This seams to be a problem with the import magic done in requests.packages. For some reasons requests.packages.chardet tries a relative import.

How to reproduce:

```
$ git clone --depth=1 https://github.com/pyinstaller/pyinstaller.git
$ cd pyinstaller
# the script runs unfrozen (well, it has an SSL-error, but no import error):
$ python tests/libraries/test_requests.py
# now freeze the application
$ cd tests/libraries/
$ ../../pyinstaller.py test_requests.py
# the frozen app fails:
$ dist/test_requests/test_requests
[...]
  File "/tmp/pyinstaller/PyInstaller/loader/pyi_importers.py", line 276, in load_module
    exec(bytecode, module.__dict__)
  File "/tmp/pyinstaller/tests/libraries/build/test_requests/out00-PYZ.pyz/requests.packages.chardet", line 19, in <module>
  File "/tmp/pyinstaller/tests/libraries/build/test_requests/out00-PYZ.pyz/requests.packages", line 95, in load_module
ImportError: No module named 'requests.packages.chardet.sys'
```

Tested with requests **version** = '2.6.0', **build** = 0x020503 and Python 2.7.9


This is a known issue in 2.6.0 with the update machinery we introduced in 2.5.2. We fixed it in 2.6.1, but unfortunately that release also broke some chunked encoding stuff. The reality is that there are no good releases right now for pyinstaller (2.1.0 through 2.5.3 are affected by CVE 2015-2296, so don't use them either), but 2.7.0 should be along shortly which will resolve this.


@Lukasa Thanks for the quick reply. I can confirm the error is gone in 2.6.1.
Make docstring of api.get more clear

I opened issue #2581, and since it is such a small change, I just went ahead and opened the pull request immediately.

This makes the docstring for `requests.get` clearer so when someone does `help(requests.get)` they can see that `params` is an optional kwarg that can be used to attach query string parameters to a GET request.  I think this change makes it symmetric with `requests.post` which specifically mentions `data` and `json` for body and json parameters.  As I mentioned in #2581, query string parameters are as relevant to GET requests as json and body parameters are to POST requests, so this change makes a lot of sense in that respect. 


Thanks for this @mhavard999, it looks great! :cake:
The docstring for `get` in `api.py` should be more clear.

This is the docstring for `get` from `api.py`:

``` python
    """Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response
    """
```

So when someone does `help(requests.get)` they won't see anything about the optional kwarg `params` used to attach query string parameters to the request.  If they're extra clever, they'll understand what "Optional arguments that ``request`` takes" mean and do `help(requests.request)`, which then explains the `params` kwarg parameter.  Note that the docstring of `requests.post` mentions `data` and `json`, even though those params are explained in the docstring of `requests.requests`.  Since query string parameters are as relevant to GET requests as body and json parameters are to POST requests, should the `requests.get` be changed to:

``` python
def get(url, params=None, **kwargs):
    """Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True)
    return request('get', url, params=params, **kwargs)
```

Current version for reference:

``` python
def get(url, **kwargs):
    """Sends a GET request.

    :param url: URL for the new :class:`Request` object.
    :param \*\*kwargs: Optional arguments that ``request`` takes.
    :return: :class:`Response <Response>` object
    :rtype: requests.Response
    """

    kwargs.setdefault('allow_redirects', True)
    return request('get', url, **kwargs)
```


Closed by #2582.
Documentation for context manager?

The documentation makes only a single reference to the fact that `requests` provides some sort of context manager, and it's in the release notes from version 0.6 in 2011:

> New persistient sessions object and context manager

Would it be possible to get examples of how to implement these context managers? Specifically, I'm looking to replace the following code:

``` python
    with contextlib.closing(urllib2.urlopen(self.url)) as dl:
```

I can't find any place that documents how to use an equivalent `requests` context manager, so I'll have to dig through the code to find out. It would be nice to get a bit more visibility into the `requests` API...


So I believe that refers to the fact that you can do:

``` py
with requests.Session() as session:
    response = session.get(self.url)
```

To get the equivalent to that code though, you could do

``` py
import requests

session = requests.Session()

with contextlib.closing(session.get(self.url, stream=True)) as response:
    response.raw.read(...)

# or

with contextlib.closing(requests.get(self.url, stream=True)) as response:
```

I could have sworn there were well documented examples of using Sessions and responses as context managers.


> I could have sworn there were well documented examples of using Sessions and responses as context managers.

Nothing I can find in the docs, only your `contextlib.closing` example from above. Feels like a good enhancement!


@sigmavirus24 Thanks for the quick response with examples. I'll try these and report back...


Update: I got the second form (the one with `requests.get` instead of `session.get`) to work. Didn't test the first form.

I had to remember to use `response.raw` as the file-like object, not `response`. (This is different from the `urllib2.urlopen()` example I gave above, where `dl.read()` gives the response). This makes sense, when I considered how the code wrapped within the context manager may want to check the response status, encoding types or other metadata, before reading the contents of the response.

Thanks!
POST Request turns into GET when prepared without a Session

The code below prints `<PreparedRequest [GET]>`

This isn't a huge issue (as I'm using `Version 2`), just kinda weird that POST is flipped to GET

``` python
import pickle
import requests
from lxml import html
from urllib.parse import urljoin

session = requests.Session()

form_url = "http://www.transtats.bts.gov/Data_Elements.aspx"

get_request = requests.Request("GET", form_url)
get_response = session.send(get_request.prepare())

page_html = html.fromstring(get_response.content)

form = page_html.xpath("id('form1')")[0]
action_url = urljoin(form_url, form.xpath("@action")[0])
hidden_inputs = form.xpath("//input[@type='hidden']")
extra_data = dict([(hidden.xpath("@name")[0],hidden.xpath("@value")[0]) for hidden in hidden_inputs ])

data = {"CarrierList" : "VX", "AirportList" : "All", "Submit": "Submit"}
data.update(extra_data)
post_request = requests.Request("POST", action_url, data=data)

# Version 1
post_response = session.send(post_request.prepare())

# Version 2
#post_response = session.send(session.prepare_request(post_request))

print(post_response.request)
```

(I'm using 2.6.2 on Arch)


This is not a bug, you're being redirected. requests automatically follows redirects, when the POST is finished you're getting a 302 to some other URL. Check `post_response.history` to confirm. =)


You're absolutely correct, thank you
Custom auth handler not called on redirect

Hi, 

I have a custom auth handler, and my problem is that if the service returns a 302 the handler needs to be re-run to generate auth headers for the request to the new url.

Right now I'm setting allow_redirects=False and handling this manually, but can this be done by requests instead?


So auth handlers tend to register themselves as `response` hooks which are called every time [`send`](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L579) is called. And that's also called when [requests handles redirects](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L188).

That said, without more details about your auth handler, I have no clue how to help you.
HEAD request not following redirects since 2.6.1

The following should follow the redirects but it just hangs

```
import requests
requests.head("http://bit.ly/DyE5", allow_redirects=True)
```

Have tested this on version 2.6.0, 2.6.1 and 2.6.2 and it stopped working at 2.6.1.

Have also tested as a GET request and that works fine on all versions.


_not_ a duplicate of #2568


Most probably a duplicate of shazow/urllib3#601
The server does not do `Transfer-Encoding: chunked` but rather an explicit `Content-Length` without a body following.


@t-8ch This is an odd dupe of shazow/urllib3#601 then, but I suppose it's possible. Definitely seems like the relevant smoking gun for now, so I'm closing to centralise there.


Ha, I was wrong. We survive the brokenness of bit.ly and die in brokenness of google.
Google seems to be generally bad on `HEAD` requests. So it definitely a dupe.

``` py
requests.head("http://www.google.co.uk", allow_redirects=True) # google.com is also broken
```

Curl also breaks.


I don't think Google is wrong, I think we're screwing up, as per shazow/urllib3#601.


But should they then not just do `Content-Length: 0` and not chunked?
After all even curl does not handle it.


Not for HEAD requests. The RFC is clear, HEAD requests never have a body, we shouldn't wait for one.


Are you sure curl doesn't handle it? By default curl won't follow redirects, it needs the -L flag.


@Lukasa ofc you are right :-)

``` sh
$ curl -XHEAD -L google.com
```

It gets another redirect with a content-length and then seems to wait for the body

``` sh
curl -XHEAD www.google.de -Lv
```

Yields no redirect for me and also hangs.

cc @bagder


I once had a chat with @bagder about the -X flag, and in particular about how it doesn't actually change curl's logic. The correct flag to pass curl is -I, and when you do that your first example works:

``` bash
$ curl -I -L google.com
```


Should have read the curl docs :-(
So we are back to shazow/urllib3#601


@t-8ch I'm sure @bagder will tell you you're not the first person to make this mistake, and you won't be the last. ;)

Yeah, that's the bug.


Yeps. The "misuse" of curl -X is widespread, even if I try to educate and document as much as I can...
requests 2.6.1 and 2.6.2 causes ResponseNotReady error for 303 answers

now I've got for redirections:

```
  File "bin/eggs/requests-2.6.2-py2.7.egg/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "bin/eggs/requests-2.6.2-py2.7.egg/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', ResponseNotReady())
```

requests 2.6 works fine.


This is a duplicate of #2568.
[Proposal] New Release Management Process

In v2.6.1 we pulled in a new copy of urllib3. Typically this doesn't negatively affect us but in this case we've seen a bunch of new issues from the chunked encoding refactor in urllib3. Because that kind of stuff _should_ not break in a `.x` release I'm proposing we follow this process when preparing releases (and that we document it for the future):

For an `v*.*.x` release, e.g., `v2.6.1`, `v2.6.2`, `v2.6.3`, etc. we _only_ include changes to requests itself. This means anything that does not live in `requests/packages/{urllib3,chardet}/*`.

For a `v*.x.0` release, e.g., `v2.6.0`, `v2.7.0`, etc. we include changes to requests _and_ update the vendored libraries. The same goes for a `vx.0.0` release, but of course those have other implications as well.


+1. Write it down in the docs, let's make it so.


:cake: Will do tonight or tomorrow morning
Keep-alive broken in 2.6.1 and 2.6.2 with chunked transfer-encoding

Starting in 2.6.1 an exception is thrown if a connection with chunked transfer-encoding is reused. I suspect that the bug is in the bundled urllib3 which was upgraded from 1.10.2 to 1.10.3.

``` python
#!/usr/bin/env python
import requests
s = requests.Session()
r = s.get('http://httpbin.org/stream/1')
r.json()
r = s.get('http://httpbin.org/stream/1')
r.json()
```

```
$ ./test.py
Traceback (most recent call last):
  File "./test.py", line 10, in <module>
    r = s.get('http://httpbin.org/stream/1')
  File "[...]/requests/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "[...]/requests/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "[...]/requests/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "[...]/requests/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady())
```


> I suspect that the bug is in the bundled urllib3 which was upgraded from 1.10.2 to 1.10.3.

As do I: can you open this bug over there, please? We'll fix it in that repository.


We should start preparing requests 2.7.0 with the fix for shazow/urllib3#598


Is there any workaround for this? I've gone back to 2.6.0 for now.


@rfortress I'm afraid not. We're working on this, there are a number of other problems with the chunked change and we'd like to land them at all once if we possibly can.

I'm genuinely sorry for how long this is taking. =(


Version 2.5.1 is latest version now? Isn't this version affected by: http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=2015-2296


The latest and affected by the bug in question, is 2.6.2. The latest that is **not** affected by the bug in question, seems to be 2.6.0 -- and it works well for me.


Correct, 2.6.0 is the version to use to avoid this bug.


I have just cut requests 2.7.0, which I believe resolves this problem (and several related ones). Thank you all for your patience!
Runtime Error

![image](https://cloud.githubusercontent.com/assets/2977844/7316104/96ea13ee-eaa6-11e4-8ee9-e64c3167c881.png)

just requests.post()

.py file is ok, but I packaged .py file to .exe file with pyinstaller in win7 32bits, and then when I ran the .exe file in a win7 64bits matchine, It was like this.


This does not like it could possibly be our problem. Do you see this if you replace the requests import with an import of any other third party library?


```
# -*- coding: UTF-8 -*-

'''
Last modified time: 2015-04-24 17:16:47
Edit time: 2015-04-24 17:17:16
File name: text.py
Edit by caimaoy
'''

import requests

if __name__ == '__main__':
    print 'aaa'
    requests.post(r'http://172.17.201.21:5000/upload', data={'test':'test'})
```

this is my code ...


Do you see this if you replace the requests import with an import of any other third party library?


Closing for inactivity.
requests (2.6.1) sometimes does not decompress gzip-encoded content

I've been trying out the DigitalOcean API when I encountered the following "problem" with requests (2.6.1):

```
token = "xxxxxxxxxxxxxx"
api = "/account/keys"
headers = {"Content-Type": "application/json", "Authorization": "Bearer "+token}
r = requests.get("https://api.digitalocean.com/v2/"+api, headers=headers, data=data)
```

... results in output for r.content and r.text like...

```
'\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x03mR\xd9\xae\xa2@\x14\xfc\x95\x9b~\xd5\x19\x96\x06\x81Nn2(^\x14\x95\xc5\x05\xc5\xc9d\xd2t\xb7\n\xb2\xc9\xa6h\xfc\xf7a\xe6y\xce\xc3IUN\xaa\xea\xa1\xce\x0bT\xd5\xe5\xf7\x95u\x15@?_ \xa2\x00)P\x82<\x1c\x82S\x94\x9dYY\x94QV\x03\x04h\x88\xa0\x8c DPA#\x01\x11\x19i<\x12\t\xc2\x1aR%D)\x12\x05D\x05\xa4\xa9(<!\x85\x81!(\x9a0\x89\xc8_\xef^\xde\xa7|++\xfc\xa1\xf73\x86\xf6\x13O\x84\x8e\x88\xd3\x7ft~\xef\xb77\xd5\x85\x8b\x95^\x07\x83\xbd(\xd3\xb9f\xc0\xe7(\xd0\x06\xd0\x89oYP\xe4\xb9\xc2\xf9\xfb\xfc\x10\xedR\xe8u#\xb8.EM.}>\xc6\xce$\x9c\x8d\x0e\xc1\xed:\xad5\xee\xacz!\x9d\xc8d\x99\x19\xb9{\xbc1ek\x8a\xc6\x05\xe3\xb83w\xd3\x91_\xeb{\xb1c\xa4\x11f>]\x96\xf1ym\xca\xde\xf3\xde\x92\xb0\xb1\xa5\xa7\x15\xef\x14\x87\x97\xdb\x96O\x9d\x9d/rT\xa2\xf1\xa2\x13\x85\xbcQ7\xb6\xd9|\xe9\x87|\x99\xa8\xae\xa4\xb1\x95\xd4\x9et%\x0e\xd2\xc3\x02{\xa2\xdb\xf8\xb2\xe9\x06\xe3k\x97\x10\xb9*\x83\x88\x8ef\x92\x15\x84\xba\x91Y\xedT\xf8:[c\xc1\xa5\x0bC\x80\xcb\xbd2\xd8\x13\xc3\xa8!\x966\xdc\xa5\xd8\x1232\xb9\xc5\xca\xa0\xeb\xc6\x9e\x901~\xda\xa9\xee5\xbc\xbcN\xac\xbbAg~\xd6\xb9\xf3\xa2VI\xf2\x9c\x04\x85\xe2\xac\x8e\xcc\x91\xac\xf95\xc8}u\x93\\v\xb6Wp\x0f\x9a?,\x0f\x87\xcb4\x0e\xdc\xaea\x99\xfe\xb5=\xcd\x99S\xb5j\xa9\xdc;\xeex\xff\xfc\xfc y\xd6\xb2G\xc5\xea\x1f\xe7\x14G\xc9w\x92\xa7}?\x19NY\xdf\xcc\xff\x8e\xef_C\x90D\xd9\xb5\x7f\x8a\xd7{\x08RV\xe3\x1e\x81:\xafq\x02\x90\xf0~\xff\x01T\xdb#\x867\x02\x00\x00'
```

... which has to be ameliorated with...

```
import zlib
data = zlib.decompress(r.content, 16+zlib.MAX_WBITS)
```

... because `r.headers['content-encoding']` happens to be `'gzip'`.

It may be good to detect gzip and other compressive encodings and proactively decompress it. Or have a default option to decompress (because in some applications, premature decompression is undesirable).

But all this seems a bit unusual because the documentation says that "_Requests automatically decompresses gzip-encoded responses, and does its best to decode response content to unicode when possible._" 

Same thing happens on CentOS Linux release 7.0.1406 (Core) (Python Python 2.7.5 via yum; requests 2.6.1) and OS X Yosemite 10.10.2 (Python 2.7.9 via Brew; requests 2.6.1).


This is a duplicate of #2561 (or #2562 - I'm on my phone and can't check). Upgrading to v2.6.2 will fix this.
Specify local port

How do I specify the local port for the socket rather than picking up a random free port?


You can do this using the [requests toolbelt](https://toolbelt.readthedocs.org/en/latest/). Get it by running `pip install requests-toolbelt`.

``` python
>>> import requests
>>> from requests_toolbelt.adapters.source import SourceAddressAdapter
>>> s = requests.Session()
>>> s.mount('https://google.com', SourceAddressAdapter(('', 6666)))
```
Merge with requesocks

Any plan to merge modification made to requests to support SOCKS proxy ?

See https://github.com/dvska/requesocks


It doesn't seem likely that we'll merge with requesocks as that hasn't been updated in over 2 years. Also that author never propsed a merge on their own.


May I propose to add this feature to requests ?


@ikus060 Many people have proposed it and even begun work: see #2425, #2156, #1982, shazow/urllib3#486, shazow/urllib3#284, shazow/urllib3#68, and shazow/urllib3#44.

You are welcome to attempt to do the work. The initial implementation should be over in urllib3: I recommend that you start work there.

If you were simply _requesting_ the feature, the core team does not believe it's an important enough priority to justify our development time. In practice, the absence of SOCKS support does not appear to be limiting the success of the library. For that reason, unless someone does the work for the implementation themselves, I do not believe we'll be implementing it ourselves.