Raibek/requests-github-issues huggingface dataset:
Response.close doesn't really close the connection

The docs for `Response.close` say: "Closes the underlying file descriptor and releases the connection back to the pool.", and the code is: `self.raw.release_conn()`.

I think this should be `self.raw.close`, as releasing the connection (https://github.com/kennethreitz/requests/blob/826667a54c608d72d8813c3eddb75b27ab26c988/requests/packages/urllib3/response.py#L121) and closing the response's underlying file descriptor (https://github.com/kennethreitz/requests/blob/826667a54c608d72d8813c3eddb75b27ab26c988/requests/packages/urllib3/response.py#L279) are different.

It's a little difficult making out all the implications, but I believe releasing the connection will close the connection _when that connection is used again from the pool_, which is different than the docs suggest.


Thanks for raising this!

This has come up before, and we don't agree with your assessment. From requests' perspective, connection pooling is A Good Thing, and we want to enable that as much as possible. For that reason, we allow urllib3 to manage the lifetime of our underlying connections, which it does very well.

From the user's perspective, the fact that "closed" doesn't mean "socket is closed" is irrelevant. What _we_ mean is that the user cannot use that raw object directly: closed is a perfectly good word for that state of affairs.

I see that you've raised this issue over at urllib3 (shazow/urllib3#379). That issue is good enough for your core problem: we're not changing this behaviour. =)


Wow your response was quick!  I'm continually impressed with this project :)

What you're saying is completely reasonable... how about removing the "Closes the underlying file descriptor" from the docs, then?  Or changing it to "Releases the connection back to the pool, which will eventually close the underlying file descriptor"?


Thanks! Keeping quick response times on this project is one of the goals. We don't accept new features often, but we want people to feel like they can suggest them, and a fast response time helps there: we don't want you to think we're ignoring you!

With that in mind, the change to the documentation is a good idea. What do you think of [this](http://docs.python-requests.org/en/latest/api/#requests.Response.close)?


Looks good to me!  Getting rid of the mention of file descriptors is probably best, given how low-level they are.
show which ssl backend we are using

This is meant to be used when triaging bugs, without requiring deep knowledge
about the internals from reporters.


LGTM, can only be helpful. =) :cake:


Did someone need this?


@kennethreitz this will help a great deal when debugging issues where people are running into errors with HTTPS connections.


Specifically #2022 would have been easier to debug if we had this information readily available to us


Hmm, I wonder if we can put this into the connection adapter instead. 


@kennethreitz I don't think we can do that. We want to know if the user installed the dependencies to make the top level import/execution work.


Of course we can, we can stick the variable wherever we want. I'm really not a fan of adding random variables around the codebase.

I think a better approach would be for the pyopenssl monkeypatcher to set a variable in urllib3, since it's the one being monkeypatched. 

Perhaps `urllib3._ssl_backend`.


Let's talk to them and see what they say. If not, we can revisit. 

I fall on the side of "we're not using pyopenssl, urllib3 is", however we're the ones manually monkeypatching, so the line is ever–so–slightly unclear. 


Yeah, I'll move it over to urllib3. On the other hand 'regular' users of urllib3 know which backend they are using, as they have to activate it themselves. Requests users get automagic.


I agree with @t-8ch that this belongs here, even if it's hidden somewhere


Just so that everyone is up to speed, urllib3 is planning to add a module that uses PyOpenSSL to 'backport' TLS features to versions of Python if it's available. This module _will_ keep track of what it's using, so we can simply wait for that to happen. Work is tracked in shazow/urllib3#371.
Unable to POST large files

When trying to POST a large binary file > 2GB, I see the following error. I've tried on multiple web severs all with the same result:

```
bash-3.2# du -h AirGuitarNation-Quicktime_H.264.mp4
2.4G    AirGuitarNation-Quicktime_H.264.mp4
....
2014-04-24 17:26:41,600 ERROR Exception caught in upload_file HTTPConnectionPool(host='REDACTED', port=80): Max retries exceeded with url: /uploader_rails/upload.py?signature= REDACTED&username= REDACTED&site_id=763&validate=localhost%3A29125 (Caused by <class 'socket.error'>: [Errno 22] Invalid argument)
```

Here is my code:

``` python
file_path = os.path.join(ROOT_PATH, 'uploads', file_name)
files = {'file': open(file_path, 'rb')}
logger.info('Starting upload for {}'.format(file_name))
    try:
        post = requests.post(upload_url, files=files)
    except Exception as e:
        logger.error('Exception caught in upload_file {}'.format(e.message))
```

Smaller files work without issue, it seems to be anything over 2GB.


Hmm, I suspect the problem here is because the multipart body is greater than 2GB in size and we try to POST the whole thing.

Want to try using the [Streaming Multipart Uploader from the toolbelt](http://toolbelt.readthedocs.org/en/latest/user.html#streaming-multipart-data-encoder)?


That's the ticket! Thank you!


Glad to be of service! =)
App Engine Error: AttributeError: 'ApplicationError' object has no attribute 'errno'

I'm not sure if this is a urllib3 error or a requests issue, but when trying to use requests on live app engine I see the stack trace:

line 55, in get
    return request('get', url, *_kwargs)
  line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  /libs/requests/sessions.py", line 452, in request
    resp = self.send(prep, *_send_kwargs)
  /libs/requests/sessions.py", line 555, in send
    r = adapter.send(request, *_kwargs)
  /libs/requests/adapters.py", line 327, in send
    timeout=timeout
  /libs/requests/packages/urllib3/connectionpool.py", line 493, in urlopen
    body=body, headers=headers)
  /libs/requests/packages/urllib3/connectionpool.py", line 291, in _make_request
    conn.request(method, url, *_httplib_request_kw)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 791, in send
    self.connect()
  /libs/requests/packages/urllib3/connection.py", line 156, in connect
    *_self.conn_kw)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/socket.py", line 560, in create_connection
    sock.connect(sa, host)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/socket.py", line 222, in meth
    return getattr(self._sock,name)(*args)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/remote_socket/_remote_socket.py", line 776, in connect
    if translated_e.errno == errno.EISCONN:
AttributeError: 'ApplicationError' object has no attribute 'errno'

This was working up until recently, so I'm not sure what happened, thanks for the help!


Thanks for raising this, but please don't raise an issue both here and on urllib3 at the same time. urllib3 and requests work very closely together and I was already looking at the urllib3 issue, so don't worry about it getting missed: we'll move the issue if necessary.


This matches shazow/urllib3#377.


Good to know, thanks for the quick response I sure appreciate it!
requests.get really slow when stream=True

I noticed that using stream=True is really slow in some cases. Code that shows the issue:

```
import requests


url = "https://api.enthought.com/eggs/rh5-64/numpy-1.8.0-1.egg"
target = "numpy-1.8.0-1.egg"

use_streaming = True

if use_streaming:
    resp = requests.get(url, stream=True)
else:
    resp = requests.get(url)

resp.raise_for_status()

with open(target, "wb") as target:
    if use_streaming:
        for chunk in resp.iter_content():
            target.write(chunk)
    else:
        target.write(resp.content)
```

With use_streaming=True, it takes around 40 sec, and only 2 sec when False. Running this script with strace, it looks like the chunk size is 1 byte:

```
...
recvfrom(4, "\345", 1, 0, NULL, NULL)   = 1 # I see many lines like this
```

Looks like for some reason the chunk size is ridiculously small ?

I am using requests 2.2.1 on python 2.7 on debian.


Hm, if I read the documentation correctly, I would have seen that the default chunk size is 1 byte... so nvm.

Is there a rationale for such a small size ?


As you've spotted, by default `iter_content()`'s chunk size is 1, ensuring that it returns as rapidly as possible. This favours responsiveness over throughput (because we endure a large number of syscalls).

Whether this is a good idea or not is unclear. We've got a giant issue that covers this (see #844), and that issue has not been decided emphatically. I'm inclined to increase the size, but wary about the risk of breaking things.

Note that the bug here is not to do with streaming: if you use `r.content` in either mode it'll be fast. It's simply the use of `iter_content` at its default chunk size that causes problems.

Anyway, the central issue is in #844, so I'll close this to centralise there.
RuntimeWarning: Parent module 'requests' not found while handling absolute import

I get this warning when running `nosetests` against a project:

```
$ nosetests
............../home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/requests/utils.py:72: RuntimeWarning: Parent module 'requests' not found while handling absolute import
  from netrc import netrc, NetrcParseError
```

However I can't seem to reproduce it in a much simpler example, so I don't know if it's some specific combination of imports or something that nose/mock is doing. You can see that test here:
- https://travis-ci.org/gds-operations/collectd-cdn/jobs/23515951#L48

Going back to v2.0.0 prior to a501b0ca81e340fb2c968e1cabc37565187006f7 doesn't produce the same warning.


Bizarre. Does this occur reproducibly when running your tests?


Yeah, everytime.

You should be able to reproduce it by cloning gds-operations/collectd-cdn@e9dc75523f77461d589ce251dc0fc662d48e1006 and running `tox`.


It's hugely confusing: it seems like requests is being removed from `sys.path`. Does that seem to be happening?


I honestly have no idea. It is at least only a warning and requests appears to otherwise work. Is there anything I can do to help debug?


What we really need is to find a minimum set of your code that reliably triggers the error. Let's start by stripping out all the tests but one that will hit that line of code and see if we can hit it.


tox uses individual virtual environments. If you don't install requests in each one then you won't be able to use requests 


> What we really need is to find a minimum set of your code that reliably triggers the error. Let's start by stripping out all the tests but one that will hit that line of code and see if we can hit it.

I'll see if I can reproduce it using `nose` (which is what TravisCI is calling directly - not `tox`, so we can eliminate that) and a much simpler module.

> tox uses individual virtual environments. If you don't install requests in each one then you won't be able to use requests

The package I'm testing has an `install_requires=['requests']` that takes care of that. The tests would fail, rather than just emit a warning, if requests wasn't available.


Closing due to inactivity.


I have requests version 2.13.0. I face this issue only when I am monkey_patching() it with eventlet's requests:
requests = eventlet.import_patched('requests')
and, when I call: r = requests.get(url)

Error:
python2.7/site-packages/requests/utils.py:113: RuntimeWarning: Parent module 'requests' not found while handling absolute import
  from netrc import netrc, NetrcParseError

This sounds like a problem with eventlet, not Requests. 

As mentioned above, the problem is in the `eventlet` library.

Here is an original tip (by @temoto) how to solve the problem: https://github.com/eventlet/eventlet/issues/208#issuecomment-76978478

And this is my workaround:

```python
from eventlet import import_patched
requests = import_patched('requests.__init__')
```

## see also

* https://github.com/requests/requests/issues/2469
* https://github.com/requests/requests/issues/2014
eval compatible repr for `Request`

After this change:
  In [1]: from requests import Request

  In [2]: eval(repr(Request('GET', 'http://x.org')))
  Out[2]: Request(method='GET', url='http://x.org')

  In [3]: eval(repr(Request('GET', 'http://x.org', auth=('hello', 'world'))))
  Out[3]: Request(method='GET', url='http://x.org', auth=('hello', 'world'))


Hey @cheecheeo, thanks for opening this!

Your code is fine and the idea is valid. The problem is that this makes the rest of the library inconsistent and it is _not_ desired behaviour. We have no intent to implement this and we have never intended for the a Request's representation to be used like this.

I'll wait for @Lukasa to weigh in, but I'm pretty sure we will not be accepting this contribution.

Cheers


Yeah, I'm with @sigmavirus24 I'm afraid. This is really nice work, but it's just not the way the requests project tends to work. Sorry! :cake:
Received "HTTP header is larger than 8192 bytes" but only given short one with requests.

Hi all, 

I'm using requests 2.2.1. I encounter a problem as follows.

In ordinary situation, it all works fine. I use requests to do some communications with server and all the header I use is:

```
{"Connection": "close"}
```

But "occasionally" I got the exception as follows:

```
  File "/share/HDA_DATA/package/lib/python2.7/site-packages/requests/api.py", line 88, in post
    return request('post', url, data=data, **kwargs)
  File "/share/HDA_DATA/package/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/share/HDA_DATA/package/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/share/HDA_DATA/package/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/share/HDA_DATA/package/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: /cate/main/3719e564ece2ad3a267ecac0f4d8db6e74333046/ (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
```

And then I check the server log, I found the error log:

```
handler.codec.frame.TooLongFrameException: HTTP header is larger than 8192 bytes.
```

It's strange, because the header is less than 8k apparently and this situation does not always happen.

Does the requests append some extra http header?

Any suggestions?

cheers,
Ivan


Requests adds a number of extra headers, but the only one that is plausibly going to be that long is cookies reflected back to the server.

I recommend using Wireshark or tcpdump to determine which header the server is complaining about. 


Ok, I think this might not be related to requests module. Thanks.


Sorry to bother again. I wanna know about what you said,  does the requests add the cookie content to the header by its own? Did I misunderstand something?

Thanks.


Requests adds any cookies that the server has sent that are applicable to the current request automatically.
[WIP] Update philosophy section

As @kennethreitz and I discussed at PyCon, this is a first pass at a rewrite of part of the Philosophy section of the docs, attempting to explain Requests' slightly unusual management style. I'd like some feedback.

Some ideas:
- should @shazow be mentioned? I felt like he should but I wasn't sure how best to explain the relationship.
- is there anything major that I've missed?


I'm like Requests' spouse.

Mrs. Requests.

#jkpleasedontsaythat


=D

I want to mention you, because I feel like you don't get enough credit. =)


❤


FYI, I made these changes a few days ago: http://docs.python-requests.org/en/latest/dev/authors/#core-contributors


@kennethreitz Yeah, I'm more interested in the description of responsibilities. =)


We should also mention here that the project is under perpetual feature freeze except at my discretion.


We can continue to iterate on this, but I'm merging for now.
Feature request: automatic auth type detection

requests could figure out whether it needs to use `digest` or `basic` auth by inspecting the `WWW-Authenticate` header.


This is an interesting idea @untitaker! It does feel like it'd be useful.

I wonder if we should trial it in [the toolbelt](https://github.com/sigmavirus24/requests-toolbelt) first, see if it's useful. @sigmavirus24?


IIRC you said (in IRC or so) that request sends the auth headers preemptively, which means in this case that we never get the chance to inspect the relevant header. I think that's something that should be addressed before implementing the feature i am requesting anywhere.


@untitaker That's not entirely true. It's what we do for Basic auth, but it's not what we do for Digest. And the fact that we do it pre-emptively for Basic is very deliberate: GitHub's API requires it. =)


Okay, in that case i can't do more than retract that comment :)


Haha, that's fine! The general idea is a good one though, we should have it somewhere.


I wonder if this would fit in with https://github.com/sigmavirus24/requests-toolbelt/pull/21

The purpose was more to manage authentication credentials for different domains but we could probably also add this case. I'd also be happy to trial this as a separate object.


@sigmavirus24
I had a similar usage to `AuthBase` in mind:

```
requests.get('...', auth=guess(username, password))
```

Where GuessAuth is something like this:

```
def guess(username, password):
    def inner(request):
        # hit network to determine auth type
        # we actually need the request object for this
        if bla:
            auth = HTTPDigestAuth(username, password)
        else:
            auth = HTTPBasicAuth(username, password)
        return auth(request)

    return inner
```

I don't think it would be nice to blend into `AuthHandler`, as the auth type
to use can vary within a domain.


Maybe the relevant parts of the guessing should be moved into another function which just returns a subclass of `AuthBase`, and not an instance (for advanced caching purposes). And if that is done, we can think about coupling AuthHandler with it.


This is being handled at the toolbelt, so I'm closing this.
ResourceWarnings

```
$ python3.4 -W "a" -c "import requests; requests.get('http://google.com')"
-c:1: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('192.168.1.2', 60578), raddr=('173.194.35.151', 80)>
-c:1: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('192.168.1.2', 36102), raddr=('173.194.35.46', 80)>
```

Same thing happens if I close() the object returned by request.get().
Unfortunately the traceback is not very helpful for determining _where_ exactly this occurs.


@giampaolo This is an artifact of our connection pooling. The socket objects aren't being explicitly closed because reusing them is cheaper than making new syscalls and reassembling the object. This is tracked in #1882.


Is there any way to tell requests/urllib3 to close the socket pool? 
Note: I need this in tests only, just to avoid the annoying warning lines to be printed.


Again, if you check #1882 you'll see that you can call `.close()` on the pool. Try this:

``` python
for pool_manager in session.adapters.values():
    pool_manager.clear()
```


Oh right, sorry. Thanks for chiming in.
auth kwarg cannot be used to authenticate an HTTPS request through a HTTP proxy

i mean so this is probably more of a rant than an issue, but that was 3 hours i won't get back.

was trying to pipe requests through a squid proxy in a lights-out environment. credentials come from a data store separate from configuration, so seemed reasonable to use requests.auth.HTTPProxyAuth(l, p) as the auth kwarg as (what i thought) a supplement to the proxies dict fed to the proxies kwarg. this works fine for http requests, but this seems to be by accident- it falls apart when making an https request through a proxy, when urllib3 uses the CONNECT verb to encapsulate the request.

basically requests.auth.HTTPProxyAuth doesn't make it clear that it's not the correct (always working) way to authenticate to a proxy, but that you must put the credentials into the proxy url in the proxies dict. deprecation/removal of that class would probably make the interface cleaner- failing that, a note in the source would be nice.


Yeah, agreed: I think we should aim to remove `HTTPProxyAuth` in version 3.0.


:+1: for 3.0 removal.


Are we still in agreement that this should be on it's way?

@nateprewitt I am in agreement, but I would be unsurprised if Cory changed his mind. Since he's on holiday, why don't you hold off on working on this so you don't waste your time.

Yep, will do. Just floating it out there for when you've both got a free moment. Thanks @sigmavirus24 :)

My mind isn't changed at this time. Let's kill it. =D

This will be resolved in 3.0.0 with #3773.
Pull latest [urllib3] to fix HTTPSConnection Redirect Loop on AppEngine

HTTPSConnection is going out as HTTP on AppEngine 1.9.0 and causing a redirect loop error.

Please pull the latest version of [urllib3].  Here is the commit for the urllib3 that fixes this issue.
https://github.com/shazow/urllib3/commit/627fc7a897e8e52b59d8ca2eb3d5ebccbe66ac7a


This is resolved and should be in 2.3.0.
Requests.get() will void SIGINT if it's in a thread

Hi, I would like to have a gentle way to shutdown the Thread by hooking SIGINT, but noticed once Requests been used in Thread, it will void Signal handler. Please test following example, and you will notice you cannot shut down by CTRL+C even the SIGINT has been hooked.

``` python
import requests
import signal
import time
from threading import Thread
import sys

def shutdownHandler(signal, frame):
    print "Shutting down..."
    sys.exit(0)

def test_load():
    x=requests.get('http://www.google.com')
    x.close()
    time.sleep(200000)


if __name__ == '__main__':
    signal.signal(signal.SIGINT, shutdownHandler)
    t=Thread(target = test_load)
    t.start()
    signal.pause()
    t.join()
```

If you comment out 

``` python
x=requests.get('http://www.google.com')
x.close()
```

Then run again, and press Ctrl+C, the process will be shutting down.
Any idea what's going in Requests? 
Thanks a lot


Thanks for raising this issue!

Requests uses httplib under the covers, which is known to have the problem of consuming SIGINT. You can see this [here](http://cuddihyd.bitbucket.org/tm/python-ctrl-c.html). Unfortunately there's nothing we can do about this directly, you'd need to fix the bug in Python itself. This bug _is_ fixed in Python 3.4.
Font layer in logo convert to curves

Some users may not have Palatino-Roman font, but in curves form they can use logo.


This looks great to me, thanks! :cake: @kennethreitz, are you happy with this?


Sorry, wrong answer in gmail. Thanks for approve!

2014-04-12 21:37 GMT+03:00 Cory Benfield notifications@github.com:

> This looks great to me, thanks! [image: :cake:]@kennethreitzhttps://github.com/kennethreitz,
> are you happy with this?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1999#issuecomment-40288137
> .

## 

Nasridinov Renat, mavladi@gmail.com


:cake:


Actually I'm going to revert this. Let's just add an SVG for that. I want a "source" version for manipulation.


Revert, I agree with you, and I'll make SVG now.
MIME type no longer set on file upload

I have the following code that was working fine with requests 1.2.0 but is not working with 2.2.1

``` python
while True:
    try: # Try to upload the image
        f = open (img_name,'rb')
        r =  requests.post(url='http://myurl.com/post', data = {'Upload':'upload'}, files = {'userfile':f}, timeout=30)
        print r.status_code
        r.raise_for_status()
        break # If the upload completes without errors break out of the loop
    except requests.exceptions.RequestException:
        print "Something went wrong, waiting and then trying again"
        time.sleep(30)
```

The code runs without errors but the image is not uploaded. The upload page rejects the post due to unsupported file type. I did a tcpdump capture while running the code with both versions of requests. The main difference I saw was that requests 1.2.0 properly sets Content-Type: image/jpeg while with requests 2.2.1 I did not see Content-Type header.

Am I doing something wrong or is this a bug?


Are you expecting the `Content-Type` header on the request or on one of the parts of the multipart upload?


I am expecting Content-Type on the second part of the multipart upload. This is properly done with requests 1.2.0


Hmm, yup, from looking at the code it's clear that we no longer guess the MIME type of the uploaded file. I wonder if that was deliberate.

Looks like it went away in af4fb8cedca7c331b8c914a40c477a2cb02055e1 (part of #1640) when we switched to explicitly using `RequestField` objects. Assuming we're happy to guess MIME types I'm happy to put it back. @kennethreitz, should we be guessing MIME types on multipart upload parts?


If it was intentional it would be nice if this would be put in the migration guide.

In trying to figure this out I saw that the mime type guessing code moved over to https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/fields.py with `def guess_content_type` but doesn't seem to be working?


It's not that it's not working, it's that it's not hit. We explicitly create `RequestField` objects, which don't do any content-type guessing.


Hey @kennethreitz I was rummaging through issues and we never did get an answer from you. Was it intentional that content type guessing was removed? There's an API for the user to specify it now explicitly so we just might want to document this.
Format doc section header

Seemed like this should have been formatted as a header (it was just
regular text) to avoid confusion.


No, it's not just regular text. See [the relevant docs section](http://docs.python-requests.org/en/latest/user/advanced/#session-objects). We want it that way: it shouldn't be its own section, it's just a useful way to highlight the value of that section.

Thanks though! :cake:


Thanks for your patience with the issues I was having with that pull request.  I see what you're saying, it just looks so different from the way it is on the webpage.  I was looking at it in terms of this:

http://imgur.com/a/7Oekj

It seems a little odd that one is regular text in that area and the other is clearly distinct from the surrounding text.  On the website it's not a header really, but distinguished, whereas on the Github docs it doesn't seem to be distinguished.


@richierichrawr That's really not a problem, we were all new once. Ours is not a community that vilifies mistakes. =)

Yeah, welcome to the fun world of GitHub. =D We use Sphinx for our documentation, which provides lots of excellent features that GitHub does not understand.
Master

Changed requests/docs/user/advanced.rst after the 1st section in that doc:

Remove a Value From a Dict Parameter

It's a distinct section from the rest of the neighboring text.

EDIT:  OK, I'm just learning about commits, and I did the edit on that doc in the browser, and now it's going to the wrong branch.  Just to be clear, I don't have anything else to commit/change other than the doc issue I cited above.  Sorry for the confusion. What's the best way to fix any problems this has caused?

EDIT2:  I figured out what happened and will submit it the right (I think) way this time.  Sorry again for any confusion as I'm just starting to learn how this works.  The submit seemed to go through on the other, so this pull can be disregarded in lieu of the other I made just now.
replace reference to crate.io

crate.io is gone, so a different mirror should be used as an example. Also, the list of PyPI mirrors should be mentioned.


This is great, thanks! Can we just update this to link to the mirrors, though, rather than calling any one in particular out? We linked to Crate because it was awesome, so now that it's gone we should just point to the mirrors in general.


We should remove all reference of the mirrors. PyPI is at the point now where the mirrors are entirely unnecessary thanks to the use of a CDN by the infrastructure team. @dstufft can speak more to this.


So it depends! PyPI itself should go down for installation very rarely now. Thanks to the fact that Fastly caches those pages for a day and if our backend servers die it'll fall back to a static mirror hosted in a different DC. Additionally one of the main reasons for the mirror infrastructure in general was the performance of downloading a file from `$wherever_pypi_is` when you live in `$place_on_other_side_of_he_world` which the CDN infrastructure more or less handles the bulk of this use case. Finally with the removal of Mirror Authenticity there is no longer any way to check that a mirror itself isn't malicous (not that we ever had anything _using_ that API to begin with) so if someone is going to use a mirror they should be selecting their own instead of just picking a random one because essentially using a mirror says "I trust you".

That being said! If Fastly itself has troubles (or more likely, one of their POP locations) then PyPI can be "down" in one region until Fastly is alerted to the fact that a region is having problems and they drain that DC. Additionally their are areas in the world where hosting a CDN is problematic. China for instance. Fastly has POP locations near China, but due to the political nature of China and their own Infrastructure any traffic that has to _leave_ China is going to take a severe performance hit because their outbound pipes are all severely congested. Fastly is unlikely to get a generally available POP location inside of China (Last I checked you could get them to manage own you own there though) because of the politics of that region would make Fastly responsible for what their customers are hosting through Fastly.

Additionally if you are going to mention a mirror it probably makes more sense to mention `https://pypi.gocept.com/` because it supports HTTPS and it's not inside China.

The tl;dr is that now adays you shouldn't generally have to ever thinking about the concept of mirrors unless you're in a problematic region like China.


Indeed, PyPi is quite solid now. When this was written, it went down daily.
Maintain DELETE method on 302

When resolving redirects, all methods except HEAD were being converted to GET to mimic browser behavior.

This change maintains the DELETE method when resolving 302 redirects.


Thanks for this!

What's your justification for making this change?


I had problems using python-swiftclient, which depends on requests in the environment at work. We had a swift cluster proxy redirecting HTTP requests to HTTPS and I couldn't delete using the client because DELETE requests were being converted to GET after redirects.

This change fixed our problem with deleting using python-swiftclient.


I should mention though that since we had problems with POST and PUT we eventually reworked our setup to use HTTPS in order to avoid this problem all together. I thought that maybe it would help someone else anyway but I'm not 100% sure that it's desired behavior without being able to do the same with POST and PUT.
Maybe it would be better to allow an optional kwarg to allow replaying requests after redirects despite the way browsers do it?


It's not the desired behaviour, sadly.

We don't want a keyword argument because it dirties the API. Fundamentally, we need to act in a way that causes least surprises, and this is it: we do the same thing with 302s as all major user-agents do. This problem caused RFC 2616 to add two extra codes that are _unambiguous_: 303 and 307.

Unfortunately, the expectation is that if you're faced with such an API you will have to handle redirects yourself, by setting `allow_redirects=False`.


Fair enough :)
Allow setting timeout on the Session

This came up in the requests IRC channel, it's something I think is generally useful (at least pip has implemented it in our Session subclass) but basically it'd be nice if the Session object had a timeout attribute that could be set to globally set a timeout instead of needing to pass it in for each request or do your own subclassing.


I'm basically in favour of this, and I'm happy to do the legwork required for it, but I want to know if @kennethreitz wants it or not.


I'd love to help out pip in anyway possible but this has been discussed twice in the last year:
- https://github.com/kennethreitz/requests/issues/1130
- https://github.com/kennethreitz/requests/issues/1563

And in both cases Kenneth was -1 on the change. It's of course likely that the needs of pip might cause a change in his mind, but given that removing that attribute from the Session object was a very intentional decision, I'm not certain what his opinion will be.


Aha, I missed those. I'm happy to assume he doesn't want it then.


FWIW I don't mind the code in pip, we have to subclass session anyways for some other stuff and the code to add timeout support isn't very large :) I opened primarily at @Lukasa's prompting :)


I wouldn't be surprised if Kenneth decided to reopen this though. =P
AppEngine HTTPS POST not working

HTTPS posts requests don't work on AppEngine.
Specifically the code here:
http://documentation.mailgun.com/user_manual.html#sending-via-api

(I removed the parts not making a difference):

```
requests.post("https://api.mailgun.net/v2/samples.mailgun.org/messages")
```

Results in a 404 (not found), which is wrong since it is clearly available.

I read a lot about requests' AppEngine support and the problems with urllib3 but I came to the conclusion that according to what I read it should work now. :)

Unfortunately I couldn't find any other service, where I can test a HTTPS Post.


If you've been reading up on our AppEngine support you know that we don't have any. =) The official position of the requests project is that AppEngine is not Python. AppEngine has got all sorts of weird stuff going on in its network libraries, and we just cannot guarantee that they're doing anything even remotely like what we're expecting they'll do.

In this case, obvious possible culprits include a lack of support for Server Name Indication causing you to talk to an entirely different host than you're expecting to. I have no idea if AppEngine supports SNI, but I'm totally prepared to believe they do not.

Anyway, the best place for this is Stack Overflow, where you might get some possible support. This is a bug tracker, and we do not consider AppEngine problems to be bugs. I'm sorry we can't be more help!


Alright, I've seen things working on Stack Overflow, so I assumed it should work. But thanks for the clear response! I already have another solution (which is by for not nearly as elegant as requests, but I'll do for now).

Haha, I like that one: "The official position of the requests project is that AppEngine is not Python.". True story. ^^
auth removes the post body

I'm using Requests 2.2.1 and when I use an auth header like so:

```
    r = requests.post(URL,
        auth=("user", "pass"),
        data=json.dumps(data)
    )
    print r.request.body
```

...there is no data in the body. However, if I remove the auth header this snippet prints out all the data in the body.


Uh, I can't reproduce this:

``` python
>>> r = requests.post('http://www.httpbin.org/post', auth=('test', 'test'), data=json.dumps({'hi': 'there'}))
>>> r.request.body
'{"hi": "there"}'
```

Can you confirm that you are actually sending data in both cases?


The body only disappears if the request makes it to the server (and I see it in my server logs). When I use the wrong password or post to a invalid URL r.request.body is set to what it should be.


I should have mentioned that my server is running behind an Apache proxy does does the auth.


Are you getting redirected?


My server returns 303s in accordance with [this API](http://docs.annotatorjs.org/en/latest/storage.html#core-storage-api), however the request body is empty when it gets to the server, so I don't think the response code could make a difference.


Wait, wait.

Do you _know_ that the request body is empty? That's a very different problem. If all you've got as proof of that is that `response.request.body` is the _last_ request. The original request body is in `response.history[0].request.body`.


I know that the request body is empty when it gets to my server from logging the request on the server.


In that case you should try to get packet capture (tcpdump or wireshark) from your server before it hits Apache. I'm finding it very difficult to believe that requests is losing the request body.


You should probably also do a packet capture on your local box to understand what's happening. Beyond checking your history on that response you should check `r.request.method`. Alternatively, you should make the same request with `allow_redirects=False`. Then check the generated request body.


Closing for inactivity.
Referrers

I have noticed that requests sessions don't send referrer links, at least in what I tried, nor have I found any mention of referrers in the documentation. The only mention of referrers in the entire request source happens in the http://hg.python.org/cpython/file/3.4/Lib/http/server.py#l1088 file.

Can anyone give information about the status of this header field?


Correct, we don't. We send the minimal number of header fields to prevent servers tripping up on what we do, but we don't realistically know what you're doing. This means that it's reckless to assert that the last page you looked up via this `Session` should be the referrer: they could be totally unrelated!

If you want to send the `Referer` header, you should send it yourself: you know better than we do. =)


Unlike a real web browser, we can not guess from which URL your request originate from. Using the previously requested URL could end up being an exposure of private data relating to what you were doing. There's no reason we should be forming or sending the Referrer header on your behalf.

Further, the issue tracker is not the best place for questions. There is an active IRC channel, StackOverflow tag, and not-so-active mailing list. Any of those three would be preferable to opening issues.
Adding socks5 support

Hey, would there be any chance to have socks4a/socks5 proxy support added to requests in the near future? For Python2 there was a module called requesocks which provided that functionality,  but apparently that one is unmaintained and has not been ported to python3 yet.

Would that be within the goals of the library?


Requests is open to having SOCKS support, but it's limited by `urllib3`, which provides lots of our underlying HTTP functionality. There's an open issue there (shazow/urllib3#284) which needs some work, and has been open for a long time: if you think there's help you can provide you should swing by and offer it!
Update urllib to 1.8 (8a8c601bee)

I noticed this version of urllib3 was attempting to use RC4 as the cipher, and SSL connections using pyOpenSSL would fail to servers that did not support RC4. Update to v1.8 version of urllib3 to get the new cipher settings pushed in https://github.com/shazow/urllib3/commit/2088570a293df42b1623dd74fcff0174d0565af5.


Thanks for this! We don't really need the Pull Request though, we update urllib3 in each release. =) I'll leave it open because Kenneth may just want to merge it anyway.


@Lukasa that is not true :)
Handle httplib IncompleteRead exception

Requests version 1.2.3

Relevant part of traceback:

```
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
    resp = self.send(prep, **send_kwargs)
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
    r = adapter.send(request, **kwargs)
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/adapters.py", line 340, in send
    r.content
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/models.py", line 594, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/models.py", line 541, in generate
    chunk = self.raw.read(chunk_size, decode_content=True)
  File "/srv/scc/virtualenvs/refinery-platform/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 171, in read
    data = self._fp.read(amt)
  File "/n/sw/python-2.7.3_slib/lib/python2.7/httplib.py", line 541, in read
    return self._read_chunked(amt)
  File "/n/sw/python-2.7.3_slib/lib/python2.7/httplib.py", line 586, in _read_chunked
    raise IncompleteRead(''.join(value))
IncompleteRead: IncompleteRead(4303 bytes read)
```


Requests v1.2.3 is several versions out of date and not supported: I suggest you try upgrading. =)


Is IncompleteRead exception handled in the latest version?


That depends. If the server has sent you less data than the Content-Length header says they have, then no.


OK, so would it be possible to add handling of that exception?


Can you reproduce it on the latest version of requests? 


@hackdna Also, what would 'handling that exception' mean? We don't know what you want to do in that situation.


Closed for inactivity.
The expected value changed for the proxies keyword

It used to be None but a recent PR changed that before my last one was merged

Fixes #1975 


:+1: Totally safe, thanks!


In related news, @kennethreitz is there an email address that the Jenkins server emails when tests fail? If not, could you add @Lukasa and/or me to the notifications? Assuming Jenkins runs on pushes to master (i.e., when you merge a PR) I would have seen this and fixed it sooner.


@sigmavirus24 I can set that up if you'd like, but it's pretty damn annoying :)


@kennethreitz It's annoying, but it's the kind of annoying that @sigmavirus24 and I are for. =D


What @Lukasa said. Also I like to know when I've broken stuff. (Because this was sort of my fault)
Testsuite Failure: test_requests_are_updated_each_time

From Jenkins:

```
============================= test session starts ==============================
platform linux -- Python 3.3.2 -- pytest-2.3.4
plugins: cov
collected 125 items

test_requests.py ...........................................................................................................................F.

=================================== FAILURES ===================================
______________ TestRedirects.test_requests_are_updated_each_time _______________

self = <test_requests.TestRedirects object at 0x2aca5b721dd0>

    def test_requests_are_updated_each_time(self):
        session = RedirectSession([303, 307])
        prep = requests.Request('POST', 'http://httpbin.org/post').prepare()
        r0 = session.send(prep)
        assert r0.request.method == 'POST'
        assert session.calls[-1] == SendCall((r0.request,), {})
        redirect_generator = session.resolve_redirects(r0, prep)
        for response in redirect_generator:
            assert response.request.method == 'GET'
            send_call = SendCall((response.request,),
                                 TestRedirects.default_keyword_args)
>           assert session.calls[-1] == send_call
E           assert SendCall(args...ects': False}) == SendCall(args=...ects': False})
E             At index 1 diff: {'stream': False, 'timeout': None, 'cert': None, 'proxies': {}, 'verify': True, 'allow_redirects': False} != {'stream': False, 'timeout': None, 'cert': None, 'proxies': None, 'verify': True, 'allow_redirects': False}

test_requests.py:1277: AssertionError
 generated xml file: /var/lib/jenkins/jobs/requests-pr/workspace/PYTHON/3.3/junit.xml 
==================== 1 failed, 124 passed in 10.27 seconds =====================
```


Fixed in b92f4ec
Add __str__ to case insensitive dict

Logging headers for debugging purposes is often necessary, and the currently logging the headers would be using **repr** which would log the implementation detail of headers, caseinsensitivedict. Adding str to case insensitive dict makes the headers of the response object more log friendly, by logging only the request itself and not implementation details.


In principle I'd be happy to take this, but you broke the build. =) Some sample output is [here](http://ci.kennethreitz.org/job/requests-pr/263/PYTHON=3.3/console): mind fixing that up?


Whoops, that's embarrassing. Fixed now!


Cool, LGTM. Thanks for this! :cake:


@sigmavirus24 Sure updated now


:+1: :cake: Thanks. Assigning @kennethreitz since @Lukasa and I agree this is ready to merge.


Will you merge this @kennethreitz ?


@avidas Kenneth is insanely busy (and probably even more so than usual as PyCon is next week), and so he does these things in batches. He'll get to it. =)


Perhaps we should just change our `__repr__`. Random split functionality is random.


Yeah, I'm going to just change our `__repr__`. Thanks, for the contribution! :cake: 

This will be a wonderful UX improvement for all of our users :)
r.url in request.post is same as request.get

In [24]: params = {
                              'f':'nopghc1xva2jkp5p6rer5j4dyr1s7r6r7b4t8j1s', #tags  
                              's':"+".join(stocklist[:10])
                            }
In [25]:
r  = requests.post("http://finance.yahoo.com/d/quotes.csv",params = params)
In [29]:   r.url
Out[29]:
u'http://download.finance.yahoo.com/d/quotes.csv?s=A%2BB%2BC%2BD%2BE%2BF%2BG%2BH%2BI%2BK&f=nopghc1xva2jkp5p6rer5j4dyr1s7r6r7b4t8j1s'
![issue](https://f.cloud.github.com/assets/6764126/2488723/116cc61c-b14a-11e3-9c22-1f86d4f821e3.png)


In requests, `params` is always query string parameters. If you want to put something in the request body, use the `data` argument: `requests.post(url, data=data)`.


thanks