Raibek/requests-github-issues huggingface dataset:
ujson option

Hi,

Is it possible to use ujson as an optional JSON encoder/decoder for requests? Or, if ujson satisfies all the use cases of requests, replace simplejson with ujson?


You absolutely can. The easiest way to do it is to monkeypatch:

``` python
import requests
import ujson

requests.models.json = ujson
```

That _should_ cause all further calls to `Response.json()` to use ujson. As that's the only place in Requests that uses simplejson, that should cover everything.


So by looking at `Response.json()`, I reckon it's safe to use `ujson` as default for `requests`?


Certainly it will be safe to override that behaviour. For obvious reasons we won't change this in the core library, but I highly encourage you to use `ujson` if it suits your purpose better. =)


Fair :)

Thank you @Lukasa !


My pleasure, I hope you enjoy using Requests! =D Let us know if you have any problems. :star:


@Lukasa - would you consider using ujson by default if it is installed?

Something like:

``` python:
# in compat.py
try:
    import ujson as json
except ImportError:
    try:
        import simplejson as json
    except ImportError:
        import json
```


Nope. =)

I see no reason for Requests to favour ujson over any other third-party JSON decoder. We do nothing very complicated with JSON decoding, so replacing the decoder we use either via monkeypatching or via doing the decoding yourself is totally safe. With that in mind, there's no good reason to move away from the standard library in Requests proper.


For those finding this issue circa-2019, please not that I believe the current way of doing this would instead be:

```python
import requests
import ujson

requests.models.complexjson = ujson
```

I've seen plenty of examples of code on Github that monkey-patches `requests.models.json`, which does not appear to accomplish the stated goal, since that attribute of `models` does not exist and is not used in the modern `requests` code base.
Plumb through new urllib3 Timeout API

From shazow/urllib3#231.

We may not necessarily want to expose the new API in its direct urllib3 form. In particular, I'm strongly in favour of keeping the current timeout API as the default usage.


I did some investigation on this today.

In principle we've got total freedom to expose any or all of the urllib3 `Timeout` class-based API. In practice, [this section of code](https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L297-L321) is a bit tricky for two reasons. First, we use `timeout` here as an argument to the queue, which _must_ be a float. Second, we don't in any sense respect timeouts for any other part of this flow.

If we leave timeouts exactly as they are now, there's no problem here and everything is fine. I don't think that's a good way to handle it though. I'll try and think about how to extend the API to behave better on chunked requests.


Looking at urllib3, it only exposes `read_timeout` and `connect_timeout`: nothing for writing data. We can limit ourselves to that as well, which makes this easy, or we can add a timeout for chunked uploads. No strong preference here.


The BDFL handled this, so I'm closing.


I have meet the same problems, and I use the version 2.10.0. But request has hangs for long time.
![image](https://user-images.githubusercontent.com/13467996/63174565-cb3ca780-c074-11e9-8fba-ed6c0813f2fa.png)

this is my request code, and it maybe is related with urllib3
remove extra mention of dangerous `r.raw`

Difference between `r.raw` and `r.iter_content` not clearly described in all places where `r.raw` mentioned. This is very dangerous, because difference not only in calling format. I purpose leave `r.raw` only when we are talking about sockets or encoded data.


@homm Thanks for this! Unfortunately, I don't agree with most of these documentation changes. My notes, in the order the change appears in the diff:
1. `stream` (or more accurately `prefetch`) was always required for `Response.iter_content`, but not always required for `Response.raw`. The portion of the documentation you're changing is about the changes introduced in the move to 1.x, and so correctly reflects the change in behaviour. This should not be changed.
2. I'm happy with this one. =)
3. Whitespace change, always fine.
4. As (3).
5. We should not be hiding `Response.raw` from people. This portion of the documentation is clear about what `Response.raw` is, so I see no reason to believe that it's dangerous. =)

If you strongly disagaree with any of those points, please let me know. Otherwise, if you remove changes 1 and 5 I'll happily merge this. =)


I understand about first, I'll bring it back. But I strongly disagree about fifth. Now it looks like `r.raw.read()` is common way to get content. As you can guess, I had the bug there. In my case I read compressed data from raw response and save it to disk.

What you think about this version?


I think the new version is much better. Change it to read "Alternatively, you can read the undecoded body..." and I'll be delighted. =D


done


Beautiful, thanks so much @homm! Would you like to add yourself to the AUTHORS file as well?


Awesome! Thanks so much for your work, keep it coming! :cake: :cookie: :cocktail:


@Lukasa I accidentally use unicode — instead of - in authors list. Sorry. Fix it please.


Heh, so you did. Fixed. =)
Wrong encoding used if charset only specified in content

HTML pages which declared their charset encoding only in their content
had the wrong encoding applied because the content was never checked.

Currently only the request headers are checked for the charset
encoding, if absent the apparent_encoding heuristic is applied. But
the W3.org doc says one should check the header first for a charset
declaration, then if that's absent check the meta tags in the content
for a charset encoding declaration. It also says if no charset
encoding declaration is found one should assume UTF-8, not ISO-8859-1
(a bad recommendation from the early days of the web).

I have a patch (pull request), more details in the commit comment.


All relevant discussion is in #1589, so I'm closing this to centralise the discussion there. =)
Add decode_content as a parameter to requests()

I would like to use Requests with Flask to make a proxy.  I must deal with some large zipped responses (and I do not care about the content).

Please add decode_content as a parameter to requests() so I can return the bytes directly back to the original caller.  Specifically, I want to use this with stream=True and pass the  iter_content() generator back to Flask for delivery to the original caller.

Thanks


Thanks for raising this issue!

This is exactly what we expose the [raw response](http://docs.python-requests.org/en/latest/user/quickstart/#raw-response-content) for. In vanilla Requests this response object is a urllib3 Response object, which contains a [`stream()` method](http://urllib3.readthedocs.org/en/latest/helpers.html#urllib3.response.HTTPResponse.stream).

The expected Requests way of handling this would be to make the request as usual with `stream=True`, and then pass `Response.raw.stream(decode_content=False)` to your Flask application. =)


@Lukasa is this documented anywhere? 


@sigmavirus24 Not as a single flow, no. It's not clear to me that it should be part of the formal documentation though: perhaps another blog post?


Documenting under Advanced Usage - Streaming Requests (http://docs.python-requests.org/en/latest/user/advanced/#streaming-requests) as just another example will hint to the reader they can use Response.raw.stream.

Thank you
websocket support

Hello:
  i was trying to figure out if there is support for websockets in requests.  Have been trying to find out but the documentation does not talk about it.  Can someone help ?  


Hi Vivek 

I believe this has been asked several times on StackOverflow. The answer is that we do not support websockets. We explicitly support HTTP/1.1 which is RFC 2675 (or something around that). It's unlikely we will support websockets in the near future too.


Websockets are not related to HTTP in any way.


So you folks dont mind if I fork and not contribute it back.


Forking won't help you in the slightest but you should read the license to answer your question. 


@vivekhub In case you are interested you can start a project like `requests-websocket` similar to [requests-oauth](https://github.com/requests/requests-oauthlib). `Requests` doesn't contain `socks` support in code base, idea of this is to have library core as small as possible but extensible.  


Good point let me first get my project done and then will figure out how to
open-source it
On Sep 7, 2013 11:07 PM, "kracekumar" notifications@github.com wrote:

> @vivekhub https://github.com/vivekhub In case you are interested you
> can start a project like requests-websocket similar to requests-oauthhttps://github.com/requests/requests-oauthlib.
> Requests doesn't contain socks support in code base, idea of this is to
> have library core as small as possible but extensible.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1580#issuecomment-24005894
> .


@vivekhub how familiar are you with ws4py? If you look at that library you'll recognize that requests and its API are not very suitable for websockets.


Also, this issue is closed, so everyone please stop commenting here.


@sigmavirus24 thanks for the pointer to ws4py looks interesting.  With my limited understanding of both the codebases it sounds like while I can start with request to upgrade a connection to a Websocket I would have to do heart surgery to get hold of the actual TCP connection which would not be the right thing to do is your concern am I right?


@vivekhub requests uses urllib3. urllib3 uses httplib for a large portion of its work and I wouldn't know where to begin getting a TCP connection out of there. I think your plan to use requests is just untenable. If you want to attempt to build an API for websockets then you're best bet is to use ws4py but I'm going to have to warn you that the requests API really seems far less than suitable for websockets.
Better Error Message When Server Uses an Older Version of SSL?

My bad, guys, this was a dupe of #1567.

Is there a way we can make a suggestion about what the issue might be in the event of a reset SSL connection? Something that clues the dev into trying an older SSL version?


Thanks for this @karanlyons! As far as I can see, this is difficult to do in this particular case. The server's response to the TLS handshake was to close the connection, which is exactly the error we report. Unfortunately, it's not clear whether that TCP RST segment has anything to do with the TLS negotiation. For instance, if the server doesn't support TLS at all you're likely to get similar behaviour.

Despite my pessimism regarding this, we do currently have a feature request open about handing this behaviour, in #1570. I'm going to close this issue in favour of centralising discussion there. =)

Thanks again for raising this issue!
Authentication with large POSTs not working (with aggressive servers)

When sending a large (multipart) POST request to a server requiring digest access authentication the "401 Unauthorized" response might be received and the connection terminated before the HTTPConnection.request() in HTTPConnectionPool._make_request() has returned causing the underlying socket routines to fail with EPIPE.

The resulting exception is not caught and the subsequent handling of the 401 response is not triggered causing the whole POST to fail with the following exception:

```
Traceback (most recent call last):
  <my code>
  File "<requests>/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "<requests>/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "<requests>/sessions.py", line 344, in request
    resp = self.send(prep, **send_kwargs)
  File "<requests>/sessions.py", line 447, in send
    r = adapter.send(request, **kwargs)
  File "<requests>/adapters.py", line 318, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host=<host>, port=80): Max retries exceeded with url: <url> (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
```

The same issue is also present when implementing essentially the same thing with urllib2.

I'm not sure if this is standard behavior for servers requiring digest authentication (have only tested the affected server so far) or if this is due to a special safe-guard against DoS attacks.

For testing I used Requests 1.2.0 and Python 2.7.4 with Ubuntu 13.04 inside VirtualBox, and I was unable to reproduce the issue on the host Windows 8.

PS. I temporarily worked around the issue by suppressing the specific exception (socket.error with EPIPE) inside Requests and letting _make_request() attempt to read the response. This worked as expected and allowed the 401 handling to kick in. But I'm sure there are some cases where this leads to undesired behavior.


@ntx Thanks for raising this issue! I agree, this is a fairly graceless way to fail. I think the correct place for this fix is in [urllib3](https://github.com/shazow/urllib3), which implements our connection-layer logic. A simple exception handler should suffice over there. Can I recommend you open an issue on that repository?
Support for HTTPS "CONNECT" method

This is a Wireshark capture of Firefox using an HTTPS proxy:

```
CONNECT ssl.gstatic.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: ssl.gstatic.com

HTTP/1.1 200 Connection Established
Content-Length: 0

<cert exchange, and other stuff>
```

Same URL, using requests:

```
GET https://sni.velox.ch/ HTTP/1.1
Host: sni.velox.ch
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: Mozilla/5.0

```

Here the proxy closed the connection, because the `https://` scheme isn't recognized by the proxy. The expected behavior is that a "CONNECT" request is sent to the proxy.

Python code used:

```
import requests

proxies = {'https': 'http://127.0.0.1:8000/'}
user_agent = {'User-Agent': 'Mozilla/5.0'}

resp = requests.get("https://sni.velox.ch/", proxies=proxies, headers=user_agent)
print resp.status_code
print resp.content

```

Traceback:

```
Traceback (most recent call last):
  File "test_client.py", line 25, in <module>
    resp = requests.get("https://sni.velox.ch/", proxies=proxies, headers=user_agent)
  File "/home/aniket/venv/py2sni/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/home/aniket/venv/py2sni/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/aniket/venv/py2sni/local/lib/python2.7/site-packages/requests/sessions.py", line 360, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/aniket/venv/py2sni/local/lib/python2.7/site-packages/requests/sessions.py", line 463, in send
    r = adapter.send(request, **kwargs)
  File "/home/aniket/venv/py2sni/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: https://sni.velox.ch/ (Caused by <class 'httplib.BadStatusLine'>: '')
```

Requests version: 1.2.3
Python version: 2.7.3


Thanks for raising this issue, @czardoz!

This is a longstanding issue that we've known about for a very long time. A quick GitHub search for "proxy https" reveals 118 issues that have been raised at one point or another about this problem. Hell, I even [wrote a blog post about it](https://lukasa.co.uk/2013/07/Python_Requests_And_Proxies/).

You'll be pleased to know that thanks to the sterling work of a few people over at urllib3, we merged a fix for this (as #1515) into our branch for the next release. This should be fixed as part of Requests 2.0.

Thanks again!


Ah, good stuff, thanks :)
Handle multiple TLS versions out of the box

There are occasional cases where we could tell a server that we also support a lesser version of TLS when connecting to them and negotiate to a lower version if necessary.

This might reduce the noise we get on the issue tracker about making requests that result in SSLErrors. A great deal of the discussion around this has already taken place over on https://github.com/kennethreitz/requests/issues/1567

I'm still trying to discern how much work would be necessary to support this but I can not image it being too much.


My understanding is that this is not how TLS works. From [Technet](http://technet.microsoft.com/en-us/library/cc785811.aspx):

> **Client Hello**. The client initiates a session by sending a Client Hello message to the server. The Client Hello message contains:
> 
> > **Version Number**. The client sends the version number corresponding to the highest version it supports. Version 2 is used for SSL 2.0, version 3 for SSL 3.0, and version 3.1 for TLS. Although the IETF RFC for TLS is TLS version 1.0, the protocol uses 3.1 in the version field to indicate that it is a higher level (newer and with more functionality) than SSL 3.0.

So we can't do this as part of a handshake. Instead, we'd have to make the initial request, fail (in some nonspecific way) and then reattempt the handshake using a lower version. That seems like a slightly crazy thing to do, especially as the number one reason for failing a handshake has got to be attempting a TLS connection to a server that doesn't support it at all.

Unless I'm wrong about any of that, I'm -1 on this. =)


This is totally unrelated, but we also need a label that says "Needs Minion Input"


@sigmavirus24 That was not a `SSLError` but a `socket.error` which doesn't really indicate a SSL problem.


@t-8ch maybe I wasn't clear but I never said that issue was caused by this, just that discussion about negotiating a different version could be found there. =)


Low hanging fruit, but could we just add a "Maybe this server needs a lesser version of TLS? {link_to_docs}" when an HTTPS connection is reset by peer? It doesn't solve the actual problem, but at least points the dev in potentially the right direction.

(FWIW, I just happened to figure out what was going on after a bunch of debugging because I searched the issues for "connection reset by peer". Not _before_ I had opened a new issue, though.)


I wonder if this should be fixed in urllib3, and then caught and rethrown in Requests. My specific concern is that urllib3 has been wrapping these in `urllib3.exceptions.MaxRetriesError`s, which makes finding the original exception just that bit more of a pain. Presumably the `urllib3.VerifiedHTTPSConnection` object can catch this failure though.

That seems like the way to go to me.


In fact, I think that's the _right_ thing to do. I don't think we should renegotiate TLS for people without telling them: that feels like a step too far. Better to raise a `requests.SSLHandshakeError` and let the user handle this themselves.


@karanlyons could you submit a PR for that documentation? You have greater context on the issue than I do.
remove ipv6 brackets from hostname for httplib

fixes #1568, makes ipv6 actually work like this:

 a = requests.get('http://[2001:718:1:4::2]/')

It might be good idea to use urrlib3 newer than
https://github.com/shazow/urllib3/commit/208ba74c796116bafc95d46dfcf4325449643ab6
as it fixes the problem too.


Thanks for this! We'll take the updated version of urllib3 though. =) Look out for the next release of Requests, which should have that updated version.
using numeric ipv6  address needs updated urllib3

using numeric ipv6 fails, as the host contains [] in name, which makes lower calls using httplib raise exception (talking about max. retries and socket.gaierror).

we need to fix local urllib copy to use self.host.strip('[]') in ConnectionPool.**init**  in connectionpool.py, or rather use urllib3 version newer than this:
https://github.com/shazow/urllib3/commit/208ba74c796116bafc95d46dfcf4325449643ab6

i will provide the small patch soon, but updating urllib3 might be better idea


Thanks for this @alekibango, and thanks for providing a PR!

It's worth noting that this was an issue we already knew about: see #1426. We work very closely with @shazow on his urllib3 project because we build Requests on top of it, and we regularly take newer versions of urllib3 with releases of Requests. I'm really glad you opened this issue, but in future it is worth quickly searching the GitHub issue tracker to see if someone else raised it before you. It would have saved you a little time! :smile:

Thanks again for this!
Don't absolute import things we've vendored.

Pointed out by @dstufft in #1560.


:+1:


Unrelated (noob) question to the pull request: Does this really make a difference? Is is better to use relative imports?


It makes it possible to use SNI when requests is vendored.


@aesptux That's an excellent question. =)

An absolute import here will search `sys.path` for a module called `requests`. That works brilliant in 99.9% of cases, which is where people have done `pip install requests`. It'll find the correct module (namely the one we're running from), and then correct find its submodules and grab the right data. All great. The time it doesn't work is when you either don't have `requests` in any of `sys.path`, or you have something else called `requests` ahead of the actual module.

The specific bug report that triggered this fix is the first one. @dstufft was building a python project that included Requests as part of its source code under a different name, to avoid clashing with a user install of Requests. He was therefore getting requests using something like `import vend_requests` or `import vend_requests as requests`. That works great for almost everything, because they were all using relative imports, so they didn't care about whether the project was actually called `requests`. This line, however, does care.

To double up the pain, the line is deliberately inside a `try...except` block that catches `ImportError`. That makes this bug particularly tough to find, as we swallow the exception that would have told you about it.


@dstufft @Lukasa Thanks for the clarification! :+1:
Support of "Transfer-Encoding: gzip"

Currently, if REQUESTS get a "Transfer-Encoding: gzip" answer, it will provide the GZIPPED content instead of the uncompressed data (like "content-encoding: gzip").

I would suggest that REQUESTS should decompress those replies, and it should add "TE: gzip" header in the requests, by default, too.


Some details: https://issues.apache.org/bugzilla/show_bug.cgi?id=52860#c5


Part of the rational for my argument against this is in the issue you provided. To quote "Tim B":

> Few mainstream browsers support gzip transfer-encoding.

requests never attempts to mimic anything that isn't in a browser and isn't very common. On the other hand, we also strive to mimic curl(/libcurl) and the fact that curl supports it (according to Tim B in the same comment) would be a point in your favor except that Tim then goes on to say:

> …however, there's no webserver that will serve this.

So I think it is safe to say that we can satisfy the largest number of our uses without it.

To not give you all of the details, however, would be dishonest of me. The Content-Encoding decompression occurs in `urrlib3` if I remember correctly. The correct place to request this would then be `urrlib3`. 

That said, I'm leaving this open so @Lukasa and @kennethreitz can feel free to correct me if I have mistated anything above.


Thanks for raising this @jcea!

This is a 'nice to have' feature: if we can get it we'll be happy, but I don't view its absence as a bug or horribly missing feature. Given that this feature belongs in [`urllib3`](https://github.com/shazow/urllib3), the connection-handling library we use, I suggest you propose it over there. If it gets implemented (or indeed if you implement it), Requests should automatically use it as well. =)

Again, thanks for suggesting this!
"requests.Session" should allow "timeout" attribute

Currently, you can set stuff like headers, authentication, etc in a "requests.Session" instance, to be used as default values for requests under that session. I would like to set a default timeout for the session too, to be used as the default timeout for requests under that session.


A timeout attribute on a Session is not a logical thing. Consider that the Session can have cookies which are represented (when sent and received from the server) as headers as well as authentication which can happen on a Session. A timeout on a Session, however, would mean that after the value assigned to that attribute the Session would stop functioning. You instead want a timeout on a Request and there is no good way to specify that. This feature has been requested frequently before (see: #1130 for example) and we are not going to add it. Attributes should be logically associative.


Indeed, we used to have this functionality, but I removed it intentially.

Thanks for the suggestion, though! It does make a lot of sense.
Include docs in source package


HI @llonchj, thanks for opening this PR! What's the rationale for doing this? I'm not 100% sold on this being a good idea.


I agree with you, that has several points of view. There's no PEP about. To
me makes sense than a source distribution includes documentation.

Kindly close the PR if not 100% sure.

2013/8/31 Cory Benfield notifications@github.com

> HI @llonchj https://github.com/llonchj, thanks for opening this PR!
> What's the rationale for doing this? I'm not 100% sold on this being a good
> idea.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1562#issuecomment-23590816
> .


@Lionchj the issue is that by default a python package does not provide any way to access the documentation post installation. Distributions like Debian and Arch package our documentation in a way that it becomes accessible but by default we have no way of doing the same in Python. If the situation were different I would be 100% for this.

Thank you for considering this though. It was extremely thoughtful and maybe if you feel adventurous you could send a message to the Distutils mailing list about this. In the meantime I'm going to close this and you can feel free to re-open it when the tools support it in a meaningful way.

Also, please don't be discouraged to send other pull requests, we're more than happy to have them.


Im -1 just to keep the file size low
SNI support doesn't support vendoring

The import in requests/**init**.py does `from requests.packages.urllib3.contrib import pyopenssl` which means it doesn't work when requests is vendored.


Thanks for raising this issue @dstufft! It's not at all clear to me why that import wouldn't work when requests is vendored, though. Can you elaborate on what the problem is?


I know what the problem is, no worries.


Just needs to be an absolute relative import.


When requests is vendored it's top level name isn't `requests`. For instance I'm vendoring it in pip and it's importable as `pip.vendor.requests`.


Yea what @kennethreitz said.


Oh, missed that entirely. =D
HTTP transport is case sensitive, it won't handle slightly malformed http URLs.

When downloading a URL whose schema is not lowercased, the connection adapter can't be found (say, for example):

_Http_ URLs.

```
Traceback (most recent call last):
  File "/Users/offby1/projects/rscp/rscp/downloader.py", line 175, in download_images
    self.download_image_url(image_url, target_file)
  File "/Users/offby1/projects/rscp/rscp/utils.py", line 45, in f_retry
    return f(*args, **kwargs)
  File "/Users/offby1/projects/rscp/rscp/downloader.py", line 200, in download_image_url
    return self._do_download_image_url(url, target)
  File "/Users/offby1/projects/rscp/rscp/downloader.py", line 208, in _do_download_image_url
    response = self.session.get(url)
  File "/Users/offby1/virtualenv/rscp/lib/python2.7/site-packages/requests/sessions.py", line 347, in get
    return self.request('GET', url, **kwargs)
  File "/Users/offby1/virtualenv/rscp/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/offby1/virtualenv/rscp/lib/python2.7/site-packages/requests/sessions.py", line 433, in send
    adapter = self.get_adapter(url=request.url)
  File "/Users/offby1/virtualenv/rscp/lib/python2.7/site-packages/requests/sessions.py", line 474, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
InvalidSchema: No connection adapters were found for 'Http://a-host.com/some-file'
```


Oh, FFS, it's fixed in devo. Never mind, obviously I've got a crufty version lying around.
Provide a quick link to http://stackoverflow.com/questions/tagged/python-requests 

It would be extremely useful to provide a quick link from inside to the requests homepage to http://stackoverflow.com/questions/tagged/python-requests

homepage: http://docs.python-requests.org/en/latest/ - see Useful Links

Currently the homepage contains 3 quick links but none of them is linking to a place where people can ask questions and get answers.

Using the bug tracker for this would be clearly against its purpose :)


Feel free to send a pull request because you're 100% correct. :)


:+1:


https://github.com/kennethreitz/requests/pull/1556


Job's a good'un! Thanks so much! :star:
Autodetect proxy on Windows

It would be nice if requests could support automatic proxy detection on Windows.

Both urllib2 and urllib accomplish this by reading windows registry via urllib.getproxies() on Windows and using _scproxy on Mac.

I opened an issue in [urllib3](https://github.com/shazow/urllib3/issues/232) but @shazow doesn't think it belongs there.


Looks looks `urllib.getproxies() is already being used (and executed).
https://github.com/kennethreitz/requests/blob/90837359632b43aba7900b65b42744715f0b739a/requests/utils.py#L416


Here's my dumb test:
1. Launch Fiddler on windows

``` python
>>> import urllib
>>> urllib.getproxies()
{'http': 'http://127.0.0.1:8888', 'https': 'https://127.0.0.1:8888'}
```

Fiddler detected.

2.

``` python
>>> urllib.urlopen('http://google.com/').read()
```

Fiddler shows request.

3.

``` python
>>> import requests
>>> requests.get('http://bing.com')
```

Fiddler shows nothing.


Ok, I'm on stable 1.2.3 and this was fixed in https://github.com/kennethreitz/requests/commit/93be6916f98f191eee9ac053994f61e99869735b#requests/utils.py

and is in the 2.0 branch, so I guess I'll have to wait for the release. Maybe you should consider 1.2.4 with hotfixes.


Hope this gets released! Specifying environment variables is tedious.
Problems with spaces in no_proxy environ variable

In utils.py, in the get_environ_proxies function:

``` python
...
    if no_proxy:
        # We need to check whether we match here. We need to see if we match
        # the end of the netloc, both with and without the port.
        no_proxy = no_proxy.split(',')
        netloc = urlparse(url).netloc

        for host in no_proxy:
            if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
                # The URL does match something in no_proxy, so we don't want
                # to apply the proxies on this URL.
                return {}
...
```

It must be replaced:

``` python
no_proxy = no_proxy.split(',')
```

With:

``` python
no_proxy = no_proxy.replace(' ', '').split(',')
```


Strictly speaking `no_proxy` shouldn't have spaces in it (it's defined as _comma separated_), but I think we can be more lenient here. =)


This should be resolved by #1552. Thanks for reporting it!
No "Host" header entry in response.request.headers

Many web servers use the Host header to match different sites, probably the most important HTTP header.

There is no Host entry in the response.request.headers, unless set explicitly when issuing the request.


@Cosmius That's a good point! I think we should fix this up for 2.0.


The "Accept-Encoding" and "Host" fields get set automatically in httplib's HTTPConnection.putrequest() if they are not set explicitly.


That's true, it does. Still, I think it'd be better to set it higher up if possible, to make the `PreparedRequest` look as much like what went out on the wire as possible. @kennethreitz may disagree though.


Actually, on second thoughts I'm really less sure about this. I'll think about it.


If we had the request object that urllib3 uses then we could perhaps pull some info out of that and put it on a yet different request object, something like `SentRequest`. `PreparedRequest` would display what work we did for this and `SentRequest` could more accurately represent what was sent over the wire. That said, I think it's quite a bit of unnecessary duplication/complication.

Also, keep in mind that when you create a request with urllib/urllib2 you never set this yourself. For even greater context, `net/http` in Ruby never does or requires this either. There is probably debugging value in this, but does the value outweigh the complexity it would cause.


Yeah, I think we're ok here. Thanks for the suggestion though!


Sorry for my absence.

I found this header interesting when I tried to reconstruct the "raw" request byte-by-byte. Things would be better if I can use req, resp = urllib2.urlopen(...)

I tried to reconstruct the "raw" request because some non-standard HTTP servers behave differently when requested with GET / HTTP/1.1 and GET http://example.com/ HTTP/1.1

This is really useful when working with the bad-designed servers.

I think you may want to reconsider it. :)


I agree that this would be useful. =)

My concern is that right now there is a working, heavily used function for inserting the `Host` header inside `httplib`. This function gets so much use that it's likely to be almost entirely bug free. To provide this header at the level of Requests we'd have to reimplement that logic. Once we do, we have a new set of code that we have to maintain, update if bugs are found, and ensure doesn't rot.

It's only worth doing that if we think there's a bug lower down that won't get fixed. As far as I can see, there is not. We have to draw a line somewhere about what we put on and what we don't, and this appears to be our line. For instance, we don't provide any way for you to see the full status line either.

When I hit these kind of problems I tend to use [tcpdump](http://www.tcpdump.org/) or [Runscope](https://www.runscope.com/), depending on the scope of the problem.
'PoolManager' object has no attribute 'proxy_from_url'

There is something wrong with the requests/adapter.py file 
![screenshot from 2013-08-22 14 58 21](https://f.cloud.github.com/assets/5285404/1007641/49f7eb30-0b0d-11e3-8ffc-9faebd93244a.png)

while using glance it shows the error:
'PoolManager' object has no attribute 'proxy_from_url'


I disagree. =) It looks like a glance problem to me. The current version of Requests does not use `proxy_from_url`. We do use `proxy_from_url` in the unreleased 2.0 branch, but it's not a class method it's a standalone function.
Release 1.2.4 to fix urrlib3

Please release new version to fix urrlib3 p3k bug: https://github.com/shazow/urllib3/issues/177

Now installation through pip is broken and I can't use dev version from Github. 


My understanding of this bug is that it doesn't impede installation at all. pip moans, but then everything goes on and is fine. The only problem is with actually using the ntlmpool functionality, which I'm not sure we expose. Are you experiencing something different?


I was just scaried from installation error: https://gist.github.com/yetty/6305114

If it doesn't mind, than it is OK.


Yeah, it's definitely scary, but it's not a problem for functionality. =) Our next planned release is 2.0, so we're trying not to rush anything out now. =) Still, thanks for raising this issue!
use a default encoding in Response's text property

why not use a default encoding in Response's text property?if the server does not set content-type explicitly, why not use a default encoding such as utf-8? the chardet.detect() function is time consuming. if some one just use response.text and does aware the inner mechanism, he may think the Requests library is too slow and change to other libraries.


Thanks for asking this question @JerryKwan! The short and pithy answer is: because it's better to be slow and correct than fast and wrong. =)

If we were concerned about speed we'd simply not have the `Response.text` property at all, and only ever use `Response.content` (with a silly hack for `Response.json()`). This avoids performing any unicode decoding at all, which will save even more time.

Having a 'default' encoding is just wild optimism, because no such default exists on the web. Saying that we'll use UTF-8 whenever we don't know what the correct encoding is means that some users will find that Requests very quickly downloads gibberish. They will then conclude that Requests, while very fast, also doesn't work properly, and they'll go and use another library. =)

**EDIT**: A user can also simulate this behaviour by searching for a `Content-Type` header with the encoding, and if it fails to find one set `Response.encoding = 'utf-8'`.


You would find that a default of 'utf-8' would make a shockingly high number of requests fail :)


Actually, I forgot to mention something. This is implemented so that you can provide your own default if you'd like.

``` pycon
>>> r = requests.get('http://httpbin.org/get')
>>> r.encoding = 'utf-8'
>>> r.text
...
```

This will fully skip encoding detection.
PEP8ified


the parts of pep8 you are using here are parts that i have intentionally chosen to ignore.

Thanks, though :)


Righto. Does that include the docstrings? I have to say that I think they are much more readable. Is there some external documentation system that needs to be taken into account?

Mind if I retry with some less controversial PEP8 choices?


Sphinx is the external documentation tool and your doc-strings shouldn't break anything but I'm not 100% sure.
[2.0] Tracking PR for the 2.0 changelog

This contains all of the current fixes merged into the 2.0 branch in changelog form. I'll keep updating this as new changes get merged.

_Note_: This uses [this link](https://github.com/kennethreitz/requests/compare/v1.2.3...2.0) as the source of changes. I'll keep consulting it as we work on 2.0.


:sparkles: :gem: :sparkles:


:beer: :beers: :beers: :beer:
Remove superfluous double dot


It's not superfluous. :-1:


It is, look at the generated docs: http://docs.python-requests.org/en/latest/user/quickstart/#timeouts


That's just plain bizarre. Perhaps it's a bug in Sphinx but I wasn't expecting that. Regardless, in the meantime I'm changing my vote to :+1:. Sorry I didn't check prior to downvoting.


This is great, thanks so much! :cookie: :star:
Allow non-string objects to be posted as data alongside files.

This should resolve #1462.


These test failures are the current cookie test failures in 2.0, nothing to do with this PR.


@Lukasa will this interfere with the undocumented streaming upload functionality?


Nope, we decide if we're doing streaming well before this point. This will only affect values in the data dict. =)


I really need to review (and document) that part of the code ;(


@kennethreitz I'm pretty sure it is documented. I'd tell you, but the docs are 503ing.


@kennethreitz Yup: https://github.com/kennethreitz/requests/blob/master/docs/user/advanced.rst#streaming-uploads
extract_cookies_to_jar requires a response with _original_response

While attempting to make a `LocalFSAdapter()` which handles `file://` urls I discovered that the response objects need to have an `_original_response` attribute or it bombs out at https://github.com/kennethreitz/requests/blob/master/requests/cookies.py#L113.


This probably doesn't help but a few things:
- Requests is ill-suited for the `file://` scheme.
- The `_original_response` object can be faked in a couple ways but only one way if you're interested in python 2 and 3 support. If the latter is your purpose then checkout sigmavirus24/betamax. I'm using the latter method there. If you have questions, feel free to ask me outside of this message.

I also wouldn't be opposed to documenting the method, but for the most part, I don't see this being a bug. Maybe @Lukasa or @kennethreitz 


Heh, @kennethreitz is the one who told me to file it :) I actually have it mocked and it's pretty janky. Simple fix for upstream though and the `file://` url is actually working pretty well, requests seems to handle it fine. The only "gotchas" are this one which requires a pretty janky hack to get around, and an error because `file://` urls don't have a hostname but another small hack can fix that too.

FWIW: https://github.com/dstufft/pip/commit/aba6f37d5619b26a4dbc4b2ceb01e9dd2899f5ad


The fix is simple without a doubt and since it seems @kennethreitz seems okay with it, I'll put together a PR with it.

That said, the best workaround is not that janky frankly but you're right in that it shouldn't be necessary. My project needs it though :/
Post not respecting explicit Content-Type in Python 3

In Python 3 (specifically, I'm on 3.3.2) if you set an explicit Content-Type header in post and have a byte string as your data, 2 Content-Type headers will get set.  Example:

``` python
    >>> _ = requests.post("http://httpbin.org/post", data=b"123", headers={"Content-Type": "application/octet-stream"})
    >>> _.json()['headers']['Content-Type']
    'application/octet-stream, application/x-www-form-urlencoded'
```

It looks like it's because the test to see if a `Content-Type` is already set is broken.  It tests (`requests/models.py`):

``` python
            # Add content-type if it wasn't explicitly provided.
            if (content_type) and (not 'content-type' in self.headers):
                self.headers['Content-Type'] = content_type
```

But keys in self.headers are bytestrings in Python 3, not strings:

``` python
    (pdb) 'content-type' in self.headers
    False
    (pdb) b'content-type' in self.headers
    True
```


Thanks for reporting this! We've actually already got a fix for this, from PR #1338, which is in the 2.0 branch. =) Should be in the 2.0 release. =)


Ah, cool.  In my defense, GitHub issue search is down :)


You absolutely don't need a defense. =) I'm glad you opened the issue: it's certainly way better than staying silent. :cake:


I've encountered this issue, too, and it makes my financial library unusable under Python 3. Is the 2.0 release imminent? Is there a workaround in the meantime?


@jaraco If you desperately need a workaround you can change the header key on L398 of models.py to be `b'Content-Type'`. That should work. I'm not expecting a huge wait for the 2.0 release, but that's really up to Kenneth.
RequestException should subclass IOError

`RuntimeError` [should only be raised if does not fall in any other categories](http://docs.python.org/2/library/exceptions.html#exceptions.RuntimeError).

Using [urllib](http://docs.python.org/2/library/urllib.html#url-opener-objects) as reference, I believe `IOError` is appropriate.

Related: https://github.com/kennethreitz/requests/issues/419


Yeah, I think you're right. @kennethreitz, are you open to me doing this for 2.0?


yes, especially if IOError subclasses RuntimeError


IOError is a subclass of EnvironmentError, sadly, not RuntimeError. It'll definitely be backwards incompatible, but I think that's probably acceptable.


could use multiple inheritence


I'm not sure that we'd be fixing the issue in that case. =D We can do it, but we're also at a stage where we can break stuff. I doubt anyone is doing `except RuntimeError` anyway, so we're probably pretty safe.


Implemented in #1536.
Request exception return encoded (windows 1255) string. how to sel locale

I'm trying to figure out how to get proper english/ascii error message from a failes `requests.get`. for some reason (probably being working with a hebrew locale), I get a jibrish unicode error message. I can deocde the message and display it correctly in hebrew, but I really prefer (for various reasons) that the Exception error text would be in english.

checking [Requests Exception source code](https://github.com/kennethreitz/requests/blob/master/requests/exceptions.py) doesn't even lead me in the right direction, Since I seem to be I'm getting a different exception:
`(10060, some encoded error text,')`
and the class is: `<class 'socket.error'>`.

So I guess we are probably looking at a `socket.error` from [socket lib](http://docs.python.org/2/library/socket.html#socket.error). At least as far as I can tell from parsing the Exception args (I can't raise the exception directly, only while running the app, where I do a smelly `except Exception as e` and parse `e.args` and `e.message`).

So: 

how do I set the locale so that requests (or the underlying socket lib if my guess is correct) would return an english/ascii error message?

Thanks for the help!


Thanks for raising this!

Unfortunately, this is not a property of Requests. I have no idea what's causing this, but you'll have to look into the socket library. If I recall correctly you also opened a Stack Overflow question on this topic, which is probably the right place to look.
bug fix for Python 3 support so that this line does not convert all dict...

... keys into binary byte strings and cause the CaseInsensitiveDict to fail to find existing keys, such as content-type


Thanks for this! Unfortunately, we've already got a fix for this in the 2.0 branch, from Pull Request #1338. Still, thanks for opening the pull request, and please keep contributing!


what is the ETA on the 2.0 branch release?  we'd like to stay current and
not have to merge this change into every release pull for a long period if
we can help it.

thanks!

On Mon, Aug 12, 2013 at 10:44 PM, Cory Benfield notifications@github.comwrote:

> Thanks for this! Unfortunately, we've already got a fix for this in the
> 2.0 branch, from Pull Request #1338https://github.com/kennethreitz/requests/issues/1338.
> Still, thanks for opening the pull request, and please keep contributing!
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1529#issuecomment-22544431
> .


We expect 2.0 to be the next release out the door, so hopefully that shouldn't be a problem. Sadly, even if it wasn't the next release out the door, we still couldn't merge this before then, because it's a backward-incompatible API change.
Raise an error for unsupported hook event

Fixes #1525 against branch 2.0
- Raises a ValueError for an unsupported hook event


Beautiful, thanks so much! =D


Hmm, I wonder if we should give people the ability to provide their own hooks...

Probably not :)


Thanks a million!


> Probably not :)

I am more strongly against it. I have no clue how we could provide 
user-created hooks in a sane way.


Well, I was thinking for use in custom adapters. Obviously, the intended actions would just be in the adapter though. So.... never mind :)


Ah that makes more sense frankly. I like that idea too... hm.