Raibek/requests-github-issues huggingface dataset:
fix resolve redirect to pass all original arguments

Fix for https://github.com/kennethreitz/requests/issues/2503
It is still a bit inconsistent but as mentioned in the issue, but it otherwise will break the current public api. 

This makes it possible for non urllib3 adapters to accept extra arguments.


I think I'd like the `kwargs` parameter in `resolve_redirects` to have a different name: `adapter_kwargs`, maybe?


I have changed the parameter.


LGTM. The one thing I'm thinking about is whether we want to allow for forward compatibility by actually storing these kwargs in a dictionary for resolve_redirects, but I haven't decided yet. I'll let @sigmavirus24 make a call here.


> The one thing I'm thinking about is whether we want to allow for forward compatibility by actually storing these kwargs in a dictionary for resolve_redirects

Could you expand on that @Lukasa? I'm not quite certain what you mean.


Any thoughts? I would like to get this merged as it is preventing me from adding a feature to my adapter (making it open source soon).


Sorry, what I meant was I'm wondering whether we should, rather than using `**kwargs` the whole way through, save the initial `**kwargs` in a dictionary which is then passed _as a dictionary_. That way, if we ever feel the need to add `**kwargs` to any of these functions later we could.

I don't think it's a big deal though.


I'm okay with this if you are @Lukasa.


Sure, why not. =)


Thanks!
Redirect doesn't pass all parameters

When you write your own adapter which takes other parameters then the default adapter, resolve_redirects only pass in the parameters which the default adapter takes, this is different from the first request as it passes all parameters to the send function.


Good spot, we should aim to fix that. Not sure where we'll store them though.


Requests currently keeps all those parameters seperatly (just in a local variables I think), instead of getting them from **kwargs invidually pass them whole kwargs to the redirect resolver? I might raise a PR for this tomorrow if I come around it.


I think having a PR to talk around would be good. I'm kinda scattered today, so it'll be nice to have something concrete to look at. =)


So I'm mildly -0.5 on this. In practice we've seen that 99.9% of the users are satisfied by how `resolve_redirects` passes parameters. So `resolve_redirects` takes exactly the parameters it needs to know about.

Having a PR is still nice to have, but I'll caution you against an immediate no-go: Do not replace the existing optional parameters to `resolve_redirects` with `**kwargs` because that breaks a public API that previously accepted positional arguments.
Kerberos auth doesn't work when the host is an alias

Hi

I'm using the request module in combination with request-kerberos. Since the server i'm querying is behind a load balancer it often happens that the authentication is done with host1 and the query on host2, leading to error 401 Unauthenticated.

I manage to workaround this problem this way:

```
host = socket.gethostbyaddr(random.choice(socket.gethostbyname_ex(host)[2]))[0]
```

Don't know if i should open this issue in `requests-kerberos` as well.


requests-kerberos should authenticate every connection so long as you attach it as an auth handler to the _session_, not to individual requests. Ideally that should avoid this problem.


Do you mean something like this?

```
s = requests.Session()
s.auth = HTTPKerberosAuth()
s.get('http://my.beautiful.alias')
```

Because this example (on their project page) doesn't work in my case:

```
requests.get("http://my.beautiful.alias", auth=HTTPKerberosAuth())
```


That is exactly what I meant. =) Does that work?


In my case it doesn't...
If it connects to the same server behind the load balancer it works, if it connects to a different one then i have authentication failed.


Interesting. Do all the endpoints share the same hostname?


The reason I ask is that Kerberos with loadbalancers is...[tricky](http://serverfault.com/questions/365065/kerberos-authentication-through-load-balancer).


That's my case:

```
➜  host judy.my.com
judy.my.com is an alias for foremanlb.my.com.
foremanlb.my.com has address X
foremanlb.my.com has address Y
```


It would be interesting to see if any other kerberos client implementation functions here. I think Chrome has a Kerberos implementation...


So HTTPKerberosAuth requires the same IP each time for authentication to continue working? I'm not sure that's the problem of requests and I'm very sure we don't want to randomly choose a value to give the authenticator if in fact that's a potential solution.


I didn't investigate deeply tbh...
Also requests need to pick up a random value and not the alias, otherwise it doesn't work.
This is the only way afaik:

```
host = socket.gethostbyaddr(random.choice(socket.gethostbyname_ex(host)[2]))[0]
requests.get(host, auth=HTTPKerberosAuth())
```

The thing is that both requests and HTTPKerberosAuth need to make their calls to the same server behind the alias.


not sure if this might help, but I had a similar problem and it seems to have gone after setting SPNs for the alias names. In my case, I had a webserver bound to an Active Directory domain with a specific URL asking for Kerberos auth. Without SPNs for the alias name in place, auth didn't work. I ran this with DomainAdmin privileges to register the SPNs:

setspn /A host/myaliashostname realhostname
setspn /A http/myaliashostname realhostname

In my case, I also had to reboot the (Windows) client machine on which I'm running the python/requests client (had problems before reboot).


Why this has been closed? It's still an issue for me


This issue is not really appropriate on this repository, it should be on requests-kerberos. =)
Add config option to use system CA bundle

I'm using requests with an internal CA which is trusted on my machines.  I recently discovered that you can modify the code in certs.py to return the system CA bundle (thus allowing verified HTTPS requests), but I think this should really be an option, not a modification to the module.

And since we all use venvs to dev, simply adding it to the Requests bundle is not a very elegant solution.


If you know the location, you can use the verify parameter on a session to set the location or you can use `REQUESTS_CA_BUNDLE` and `CURL_CA_BUNDLE`.

We used to have a way to auto-discover system bundles but there were far too many corner cases.


This is another thing that can be solved by shazow/urllib3#507, if it gets merged. Until that time, I agree that this is more complicated than simply passing your own certificate bundle to requests.

For anyone who is interested, you can build this bundle by running:

``` bash
$ cat `python -c 'import requests; print requests.certs.where()'` <your PEM file>
```
Passing multiple certs for CA validation

It would be nice that requests allowed passing multiple certs for CA validation. It might be for reason or another be unfeasible to merge CA certs into one bundle. In comparison stdlib ssl you can run load_verify_locations arbitrary amount of times to load more certificate locations. 


Requests absolutely does allow that, you just need to concatenate your various certs together into one file that you then pass to `verify`.


Yeah, I was meaning without concatenating your certs together in one file. That's specifically what load_verify_locations called multiple times avoids. That's especially useful on Windows since the system certificate store is not a file but a binary storage accessed through an API.


@nanonyme Currently we can't achieve this: we just don't have the primitives. At the very least we would need shazow/urllib3#507 to get merged before this is even possible.

Additionally, the stdlib SSL module only allows this for some versions: specifically, those with SSLContext objects. That limits it to 2.7.9 and later, which is problematic.

Finally, of course, this doesn't solve the Windows problem since Windows users can't point OpenSSL at a file _period_, they need another solution.

Altogether, I think this is just too niche a request to satisfy at the moment. It seems like it would make life easier for a tiny minority of users, but that minority would be just as well served by a merge of sshazow/urllib3#507 and then playing with SSLContext objects.

Thanks for the suggestion, though! Please keep them coming. =)
Cookies + proxy

I'm sending requests to some site using proxy servers for each request another proxy.
I'm not a pro in this sphere, so I'm interested in a such things:
- are cookies generated each time while i'm sending requests?
- can site (to which i'm sending request) identify my computer based on cookies or another factor and to find out that this request and other ones where made especially from my computer?


1. It depends. If you aren't using a [Session object](http://docs.python-requests.org/en/latest/user/advanced/#session-objects), then yes, cookies are generated each time because we have nowhere to store them. If you _are_ using a Session object, then you should re-use the cookies, because cookies are tied to the origin server, not the individual connection.
2. Yes. That's exactly what cookies are for.


@ttomoday By the way, while I did answer your questions, please note that this issue tracker is a place for reporting bugs and discussing improvements, not for general questions about library usage or about HTTP more widely. I recommend using Stack Overflow to ask further questions of that nature. =)


okay, thanks for your answers
made the quickstart more reader friendly

Closes #2418


Can we choose a better header to set? Or a better value? I can't see how a long user-agent string will be reader friendly. Some people will now start asking why all that information is necessary and what it means and why it's important. Why not use `headers={'Connection': 'close'}`?


Or just shorten the user-agent. `User-Agent: my-app/0.0.1` should do just fine.


Made the string shorter.


Hurrah! :cake:


Proud of my first contribution to requests :+1:  Here's a :cake: for everyone :smile: 


@yasoob Feel free to open a pull request adding yourself to AUTHORS.rst!
Allow some way to implement monitor callbacks for files in POST requests

I am currently developing a small tool to download and upload files. I am using the requests and progressbar libraries, among others. During downloading and uploading of files I want to display the progress and speed of the upload. This works very well with the download, I simply set stream=True in my request and then use iter_content on the Response object with some chunk_size and report the total loaded bytes back to the progressbar.

As far as I can tell this is impossible with files sent via POST requests. I am trying to upload a single file by passing a dict as "files" to the POST request and would like to somehow implement a monitor callback that would either report how many bytes of that specific file have so far been uploaded, or how many bytes of the entire request have been uploaded -- either way works.

I checked the requests internals, and the magic seems to happen in: request.models.RequestEncodingMixin._encode_files. The problem is that in the last few lines of that function, the data always seems to be read into memory all at once, making it impossible to decorate or implement the read method of the file-like object in such a way that would allow monitoring the current upload further down the line.

Is there some kind of way to do what I am trying to do, and if there isn't yet, would you consider adding it? Maybe this would fit neatly into the hooks-system which doesn't seem to have that many features as of yet...

EDIT: The [requests-toolbelt](https://toolbelt.readthedocs.org/en/0.3.0/user.html#monitoring-your-streaming-upload) seems to have the solution.


For others looking at this, we discussed this on IRC and worked out the correct solution using the [requests-toolbelt](/sigmavirus24/requests-toolbelt).
Need documentation about "Cookies set on individual Requests through a `Session` are not persisted to the session"

It took me a while to figure it out. Thought it might be a bug initially, I then came across #1791 & #1728.

Since this feature by design is quite confusing. I feel it deserves some more attention apart from simply mention it in the changelog (0b68037).

Can we have it as a tip/hint some where in the documentation (or quick start)? And suggest the proper way to prepare the persistent cookies.


Sorry, it's late here and I'm tired. Could you give an example of what you mean?


So I think this is a good idea. It's not entirely clear that we don't send the custom cookies in the second request here:

``` python
s = Session()
s.get('http://myhost.com/page1', cookies={'from-my': 'browser'})
s.get('http://myhost.com/page2')
```

I'll happily merge any pull request that adds documentation for this. =)
Import urllib3's Retry location from the right place

Importing from urllib3's top-level location causes the namespace to be
urllib3.util.retry.Retry instead of
requests.packages.urllib3.util.retry.Retry. Without this fix, an using
requests with an un-vendored version of urllib3 will break when urllib3's
retry handling kicks in.

Closes shazow/urllib3#567


@Lukasa feel free to merge this so we can release 2.5.5 tomorrow.


:cake:
Sanitise verb string. Now we can use unicode strings in http methods.

There is error (`UnicodeDecodeError: 'ascii' codec can't decode...`) when use unicode string as http verb in instance Request.


Interesting.

Why do you want to do this? RFC 7230 is pretty unambiguous about what's allowed in a HTTP METHOD: namely, `token`. The ABNF defines `token` as:

```
 token          = 1*tchar

 tchar          = "!" / "#" / "$" / "%" / "&" / "'" / "*"
                / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~"
                / DIGIT / ALPHA
                ; any VCHAR, except delimiters

 DIGIT          =  %x30-39
                        ; 0-9

 ALPHA          =  %x41-5A / %x61-7A   ; A-Z / a-z

 VCHAR          =  %x21-7E
                        ; visible (printing) characters
```

That would seem to guarantee that any valid method is necessarily representable in ASCII. So what's the rationale for allowing non-ASCII methods?


I agree with you about RFC 7230. But documentation does't have mention that http verbs must be in ASCII, there are only list of allowed verbs. Maybe better clarify this part.


That seems like a reasonable decision to me. I'll happily accept any pull request that updates the documentation (probably by updating the docstrings in `requests.request`, `requests.models.Request`, and `session.request`).
Consider adding return type documentation in docstrings

It would be great if the docstrings of (at least) the top level entry points (`request`, `get`, `post`, etc.) could define the return type in a sphinx compatible way (`:rtype:` http://sphinx-doc.org/domains.html#info-field-lists). This would make it easier to work with requests in various IDEs and code inspection tools.


Sounds good to me!

Do you want to open a pull request containing this change?


Sure. I'll try to fit it in later in the day.


:+1: 


Looks like this was fixed by #2487. Closing as fixed.
Update urllib3 to 43b5b2b452e4344374de7d08ececcca495079b8d


```
(requests)~/s/requests git:update-urllib3 ❯❯❯ py.test                                                                                                                                                                                                       ✭
==================================================================================================================== test session starts =====================================================================================================================
platform darwin -- Python 2.7.6 -- pytest-2.3.4
collected 156 items

test_requests.py ............................................................................................................................................................

================================================================================================================ 156 passed in 24.73 seconds =================================================================================================================
(requests)~/s/requests git:update-urllib3 ❯❯❯ python                                                                                                                                                                                                        ✭
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get('https://httpbin.org/response-headers', params=[('WWW-Authenticate', 'digest'), ('WWW-Authenticate', 'basic')])
requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r.json()
{u'Content-Length': u'126', u'Content-Type': u'application/json', u'WWW-Authenticate': [u'digest', u'basic']}
>>> r.headers
{'content-length': '126', 'server': 'nginx', 'connection': 'keep-alive', 'access-control-allow-credentials': 'true', 'date': 'Wed, 11 Mar 2015 00:38:36 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json', 'www-authenticate': 'digest, basic'}
```


:+1:


I'm going to tag 2.5.5 as a quick fix. Then we'll start working on 2.6.0
Radio streaming download: ICY causes BadStatusLine

Hello,

Even if the icy's protocol is broken, I would still need to download the streaming data of a shoutcast server.
When attempting to, I get this issue:

  File "/usr/lib/python3/dist-packages/requests/api.py", line 55, in get
    return request('get', url, *_kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='streaming3.radiocat.net', port=80): Max retries exceeded with url: /;listen.mp3 (Caused by ProtocolError('Connection aborted.', BadStatusLine('ICY 200 OK\r\n',)))

Any workaround or something I could do?

Thanks.


Yes. =) Don't use a HTTP library.

That answer was a bit glib, but it's pretty accurate. Let me show you what RFC 7230 says a HTTP status line should look like:

`status-line = HTTP-version SP status-code SP reason-phrase CRLF`

That first set of characters needs to be a HTTP version. If it doesn't specify a HTTP version, a HTTP library cannot assume it has the faintest idea what's going on. For that reason, `httplib` sees a HTTP version called 'ICY', doesn't recognise it, and bails out (quite rightly).

Unfortunately for you, `httplib` then closes the socket connection and loses its reference to the socket object.

What you really need is something that can fake out a HTTP request using a socket. This shouldn't be very hard: create a socket connection directly to your target and then just write some basic HTTP headers to it.

If you find that helpful, you could write an ICY transport adapter for requests and use it.

Sadly, however, this is outside the scope of requests to help you. =)


Lukasa, thank you for your answer.

How bad would it be to modify the original library like this?
https://mail.python.org/pipermail/web-sig/2005-August/001641.html

It sounds like I'd save a lot of work just by adding two lines. Still, I know it's not a great solution. What do you personally think about it?

Doing everything with sockets will be a lot of pain.


@rubentorresbonet if you choose to modify httplib like that, that's your decision. It's not up to our discretion nor is it our place to recommend it.

I think modifying a standard library is a poor way of handling this, but if that's what you choose to do, there's nothing for anyone to do about it.

Also @Lukasa didn't tell you to do _everything_ with sockets, only portions dealing with ICY.
Returning json from SocketServer: ConnectionError BadStatusLine

This worked in requests version 2.2.1, but breaks in 2.5.3
Environment:
Ubuntu 12.04/x86
Python 2.7.3


@manojgudi can you give us more information. We have absolutely nothing to debug this with otherwise and will be forced to close the issue.


@sigmavirus24 Thanks for prompt reply! Lemme write a script to reproduce this issue.. Gimme a day or so..

Thanks


Turns out I didnt attach http headers when returning a json response from my custom SocketServer(which was causing this issue) 
sorry for the noise..
Document Response.iter_lines() deficiencies


@plaes This is a great PR. One small note however, we don't use `unittest`, we use `pytest`. This is why your test suite failed: `unittest.expectedFailure` doesn't exist in Python 2.6. Instead, use [`pytest.mark.xfail`](http://pytest.org/latest/skipping.html#mark-a-test-function-as-expected-to-fail).


Oops.. I push forced the fix.


Hurrah! Thanks so much for this @plaes!


:cake: :star:


Thanks @plaes!
Make Response.iter_lines() reentrant safe

Calling r.iter_lines() multiple times doesn't lose pending data between
calls to this method.


This is fix for issue #2476. So either this, or documentation should mention that Response.iter_lines() is not reentrant safe.


Eh, fine by me. @sigmavirus24?


I feel likewise "Eh". This is just going to be broken if there aren't any tests for it. It also breaks the thread-safety of a response I think. (I would hope someone's not consuming a response from multiple threads, but this makes the iterator less threadsafe.)


Yup, this approach is broken. I'll see whether I can find a way to fix it.


I'm closing this for now.. and submitted documentation about this issue and testcases in another pull requests.
iter_lines: Skipping first line using next(..)

When trying to skip first line of  `response.iter_lines()` using `next()`, some extra data from following lines is also consumed.

Test script:

``` python
import requests
from contextlib import closing

with closing(requests.get('http://httpbin.org/stream/5', stream=True)) as r:
    skip = next(r.iter_lines())
    print 'SKIP: ', skip
    for n, l in enumerate(r.iter_lines(), start=1):
        print 'PROCESS: {}'.format(n), l
```

With following output (line 1 totally missing, and line 2 being partial):

```
SKIP:  {"url": "http://httpbin.org/stream/5", "headers": {"Host": "httpbin.org", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "User-Agent": "python-requests/2.4.3 CPython/2.7.9 Linux/4.0.0-rc2+"}, "args": {}, "id": 0, "origin": "xxx"}
PROCESS: 1 n.org/stream/5", "headers": {"Host": "httpbin.org", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "User-Agent": "python-requests/2.4.3 CPython/2.7.9 Linux/4.0.0-rc2+"}, "args": {}, "id": 2, "origin": "xxx"}
PROCESS: 2 {"url": "http://httpbin.org/stream/5", "headers": {"Host": "httpbin.org", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "User-Agent": "python-requests/2.4.3 CPython/2.7.9 Linux/4.0.0-rc2+"}, "args": {}, "id": 3, "origin": "xxx"}
PROCESS: 3 {"url": "http://httpbin.org/stream/5", "headers": {"Host": "httpbin.org", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "User-Agent": "python-requests/2.4.3 CPython/2.7.9 Linux/4.0.0-rc2+"}, "args": {}, "id": 4, "origin": "xxx"}
```


This is a tricky interaction with iterators. Most iterators don't quite behave the same way. What you need to do is the following

``` py
with closing(requests.get('https://httpbin.org/stream/5', stream=True)) as r:
    iterator = r.iter_lines()
    print 'SKIP: ', next(iterator)
    for n, l in enumerate(iterator, start=1):
        print 'PROCESS: {}'.format(n), l
```

That works fine for me. I'm not confident this is a bug because `iter_lines` is allowed to make some assumptions about how it's used. I'll wait to see if @Lukasa agrees with me.


Thanks, this was unexpected :)


Agreed.

`iter_lines` and `iter_content` should _really_ only be used in a single place, or they risk having unexpected effects. It might be worth adding a `..warning` or `..admonition` block to the docstrings for those methods to reinforce that message.


So I took a quick look at the implementation of [`Response.iter_lines`](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L693)

We read a chunk from the response, and then find the line within that chunk and return it. We store the remainder locally in the function and not on a the response object itself. I think an update to the docs like @Lukasa described is warranted. 


Would it make sense to actually store the pending data as `self._pending`?
Use Requests in Pycharm has a ImportError

### Error: AttributeError: 'module' object has no attribute 'get'

When I use Pycharm Requests, error: "AttributeError: 'module' object has no attribute 'get'"
I modified file "requests\packages\urllib3\connection.py" in the 8-11 row

``` python
try: # Python 3
     from http.client import HTTPConnection as _HTTPConnection, HTTPException
except ImportError:
     from httplib import HTTPConnection as _HTTPConnection, HTTPException
```

To:

``` python
py   = sys.version_info
if py >= (3, 0, 0):
    from http.client import HTTPConnection as _HTTPConnection, HTTPException
else:
    from httplib import HTTPConnection as _HTTPConnection, HTTPException
```

It worked.
### Environment：

Pycharm 4.04
Python 2.75


What is PyCharm doing here? Because it seems to be totally confused.

This is a really common pattern, and it's considered good style: it's better to test for features than to do strict version checks wherever possible. The follow-on is: there's never access of an attribute called 'get' from the module you changed. So again, it feels like PyCharm is getting this wrong.


No response in almost a month. Closing as inactive.
requests 2.5.4 mishandles WWW-Authenticate headers

Executing a get() with requests 2.5.4 I now stumble over an unexpected 401 error.
Looking into it I see that response.headers contains:
{'server': 'Microsoft-IIS/7.5', 'content-type': 'text/html', 'date': 'Thu, 05 Mar 2015 11:56:25 GMT', 'content-length': '1293', 'x-powered-by': 'ASP.NET', 'www-authenticate': 'Negotiate'}
and the HttpNtlmAuth handler installed onto my session object has not kicked in.
I have a feeling the "WWW-Authenticate: NTLM" header which is also in the response disappeared somewhere:
![headers](https://cloud.githubusercontent.com/assets/220082/6504440/b22c5680-c337-11e4-89bb-6f2da348c9fd.png)


Note the above works as expected after downgrading to requests 2.5.1.


Thanks @dieterv, this is a known issue, see shazow/urllib3#561. We aim to push out a fix for this very shortly.
requests can't detect the right charset

I write some code in python 3:

```
url = "http://www.kanunu8.com/book/3981/42775.html"
r = requests.get(url)
print(r.encoding)
print(r.text)
```

from the html source code,it clearly state it's charset is gbk,but r.encoding is "ISO-8859-1".
BTW,I can get the right html from urllib.request.


Hi there!

This is a well known requests behaviour that is clearly documented. I answered this in a previous GitHub issue [here](https://github.com/kennethreitz/requests/issues/2122#issuecomment-48448778): please consult that answer.


But the function`requests.utils.get_encodings_from_content` raise a exception

```
Traceback (most recent call last):
  File "D:/Developer/Python/PythonEx/Http/Crawler/requests_with_coding.py", line 5, in <module>
    codings = get_encodings_from_content(r.content)
  File "C:\Python34\lib\site-packages\requests\utils.py", line 304, in get_encodings_from_content
    pragma_re.findall(content) +
TypeError: can't use a string pattern on a bytes-like object
```


We plan to remove `get_encodings_from_content` in requests 3.0, because we don't think we should have any code to support reading HTML at all. We are _not_ an HTML library, we're a HTTP library, and we don't want to introspect request bodies at all if we can help it.

The code for it is extremely simple, you can easily bring it into your own project. Find it [here](https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L288).


I got it, thanks a lot :)


For anyone may see this, you can use Beautifulsoup to get the desired html:
just use: `soup = BeautifulSoup(r.content)`
Avoid data duplication when creating a Request with str/bytes/bytearray input

I've worked on the idea discussed in #2468 regarding avoiding duplicating the file data when creating a Request with files input that is str/bytes/bytearray and not a stream. The code currently creates a BytesIO or StringIO stream using the str/bytes/bytearray as initial input. This is then read out with fp.read() when creating a RequestField. The problem with this is that BytesIO and StringIO creates a copy of the input. Thus, if you have created a 100 MB file in memory using a bytearray which is used to create a Request, three identical copies is produced:
_(1)_ The original bytearray, _(2)_ The BytesIO/StringIO object, which has its own copy, and _(3)_ The copy that is read out with fp.read() and stored in the RequestField object.

If we instead of creating BytesIO/StringIO objects just pass the str/bytes/bytearray on to RequestField, we avoid making the two duplicates of the data. This yields a **30-40% performance improvement** for files larger than 5 MB when creating a Request. The end result is, as far as I've been able to tell, the same.

I have created a GIST with my code for checking that the proposed change yield the same body as the old. It also has the code for evaluating performance and the results of the three ways I've compared the performance differences. https://gist.github.com/scholer/57fced2b63d1bb3130eb

The tests in test_requests.py completes without failure on python 2.6 and 3.4. I don't believe any new tests are required, but let me know.

**Question:** Is this really relevant? Why would you ever build a large file in memory? Large files are always read from disk or handled as some other stream.
Answer: Some cloud storage services, e.g. Amazon AWS, relies on adding a "prefix" and "suffix" to the file content, which includes e.g. upload authorization tokens. You can build the file data using e.g. a BytesIO stream or a bytearray. The bytearray can be used through a memoryview, which supports Python's buffer protocol. This means that the file data can be inserted directly into the bytearray without extra memory overhead.

```
fp.readinto(mv[offset:offset+length])
```

To my knowledge, it is not easy to read the content of one stream directly into another. For instance, appending a BytesIO with filecontent using  fp.readinto(bytestream) # will fail since bytestream does not support the buffer interface

Note: Although my tests says that the final body is the same with the new code as the old, I would still recommend that someone with more experience takes a look at this to make sure that it does not have any adverse effects.

Thanks.


Thanks for this!

I can see no particular reason to be concerned about this patch, it seems totally reasonable to me. I'll obviously wait for @sigmavirus24 to hop in. It would also be extremely helpful to get someone who is deploying requests in anger to weigh in here.


This looks fine but I'd like to test it a bit. Sorry for the delay. Too many things to do
Unable to import requests

Hi on [Travis](https://travis-ci.org/wikimedia/pywikibot-core/jobs/52812789#L742) our bot is unable to import requests 2.5.3 properly on Python 2.7 and 2.6:

```
  File "/home/travis/build/wikimedia/pywikibot-core/.eggs/flickrapi-2.0-py2.7.egg/flickrapi/__init__.py", line 52, in <module>
  File "/home/travis/build/wikimedia/pywikibot-core/.eggs/flickrapi-2.0-py2.7.egg/flickrapi/core.py", line 13, in <module>
  File "/home/travis/build/wikimedia/pywikibot-core/.eggs/flickrapi-2.0-py2.7.egg/flickrapi/tokencache.py", line 10, in <module>
  File "/home/travis/build/wikimedia/pywikibot-core/.eggs/flickrapi-2.0-py2.7.egg/flickrapi/auth.py", line 23, in <module>
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/__init__.py", line 19, in <module>
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/adapters/__init__.py", line 12, in <module>
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/adapters/ssl.py", line 12, in <module>
    SSLError -- exception raised for I/O errors
  File "/home/travis/build/wikimedia/pywikibot-core/.eggs/requests-2.5.3-py2.7.egg/requests/__init__.py", line 53, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/tmp/easy_install-uogHkP/requests-2.5.3/requests/packages/__init__.py", line 49, in load_module
```

It seems that this only happened recently. [This run](https://travis-ci.org/wikimedia/pywikibot-core/jobs/51751467#L478) worked fine with 2.5.1.

Locally I could install flickapi (2.0) and requests (2.5.3) and import them without problems on Python 2.7.8.


Thanks for the report!

This is a known regression in 2.5.2 and 2.5.3, with a fix being tracked under #2466.
RuntimeWarning when using eventlet for patching requests

Hi,
I am getting the following warning while using requests with eventlet:
/usr/local/lib/python2.7/site-packages/requests/utils.py:74: RuntimeWarning: Parent module 'requests' not found while handling absolute import from netrc import netrc, NetrcParseError

looking in the issue below that have similar warning
https://github.com/kennethreitz/requests/issues/2014
could it be that it related to the fact (if I am not wrong) that during the eventlet patching,  requests removed from the sys.path.

from the eventlet docs:
http://web.archive.org/web/20110723123938/http://www.eventlet.net/doc/patching.html#monkeypatching-the-standard-library

eventlet.patcher.import_patched(module_name, _additional_modules, *_kw_additional_modules)
Imports a module in a greened manner, so that the module’s use of networking libraries like socket will use Eventlet’s green versions instead. The only required argument is the name of the module to be imported:

import eventlet
httplib2 = eventlet.import_patched('httplib2')
Under the hood, it works by temporarily swapping out the “normal” versions of the libraries in sys.modules for an eventlet.green equivalent. When the import of the to-be-patched module completes, the state of sys.modules is restored. Therefore, if the patched module contains the statement ‘import socket’, import_patched will have it reference eventlet.green.socket. One weakness of this approach is that it doesn’t work for late binding (i.e. imports that happen during runtime). Late binding of imports is fortunately rarely done (it’s slow and against PEP-8), so in most cases import_patched will work just fine.

so I just want to verify the the warning is ok (because looks that the patched version of requests works quite fine)


eventlet shouldn't be affecting the import of netrc. I'm also not certain if eventlet has special case handling for requests. All eventlet should be patching is socket or httplib (or both) as far as I know, so this seems like a bug for eventlet, not us. I don't use eventlet with requests, nor does @Lukasa so neither of us can tell you that this isn't going to break anything (i.e., that the warning is okay). So if you want assurances, you'll need to search elsewhere (e.g., ask a question on StackOverflow and tag it with eventlet so that someone more familiar with eventlet will answer your question).


As mentioned above, the problem is in the `eventlet` library.

Here is an original tip (by @temoto) how to solve the problem: https://github.com/eventlet/eventlet/issues/208#issuecomment-76978478

And this is my workaround:

```python
from eventlet import import_patched
requests = import_patched('requests.__init__')
```

## see also

* https://github.com/requests/requests/issues/2469
* https://github.com/requests/requests/issues/2014
Only add VendorAliases for vendored dependencies

Closes #2465 
Closes #2470 

/cc @dstufft Let me know if I should send this to pip as well


I was doing some other work and a few corner cases popped into my head that this breaks. Previously the following would break

``` py
from requests.packages import sys
from requests.packages.chardet import os
```

Now they'll happily succeed. 


I would be so much more comfortable if we weren't relying just on pip's tests, but take a look at [this run](https://travis-ci.org/pypa/pip/builds/52597911) (see also: https://github.com/pypa/pip/pull/2465) to see that this works the same way it did before, but now with the added benefit of not allowing imports like `from requests.packages.chardet import sys`.

The import error message will vary though depending on whether or not `urllib3` or `chardet` are vendored though. Perhaps we care, perhaps we don't.


Also many thanks to @dstufft for helping with the final solution.


@ralphbean @eriol if you two wouldn't mind testing this on Fedora and Debian respectively, I'd greatly appreciate it. (Alternatively, if y'all have scripts that I could test this with on the respective distros, I can spin up cloud images to test with.)


@Lukasa should I just merge this myself to push 2.5.4 out the door?


Yes, do it.


Looks good to me.  Sorry for the late reply.


It's look good also to me. Sorry for my late reply I was a little busy.


No worries. I think I'll release 2.5.5 today to include this patch for Python 2.x users
Multiple requests suggested way

Hi,

I am trying to do multiple requests without having to wait for each one to finish. I was using grequests but i am wondering if there is a better way to handle it.


grequests is the best way to handle it at the moment if you like using gevent. Your other options are [trequests](https://github.com/1stvamp/trequests), [treq](https://github.com/dreid/treq), [async-requests](https://github.com/inglesp/async-requests), and there are likely some others floating around that I don't know about.
Ampersand (&) added for no reason?

Hi guys, could you confirm if this is a bug or works as intended?

``` python

import requests
import json

url = "http://server:8080/jsonrpc?request=";
headers = {'content-type': 'application/json'};
payload = {blah};

r = requests.get(url, params=json.dumps(payload), headers=headers);

print(r.url);

```

It outputs the encoded string BUT for no reason adds an ampersand as below:

http://server:8080/jsonrpc?request=**&**{blah}

This provokes parse errors when querying XBMC/KODI server.

Thanks!


It is indeed working as intended.

The problem is this: we have no real way of knowing what you mean when you provide part of the query string as part of the URL and part in the 'params' section. You're doing something particularly unexpected by passing parameters as a string rather than a mapping (though I suspect that's for the sake of this limited example and not your actual code).

The position is generally that if you want us to build your query string you should let us build the whole thing, or at least build whole name-value pairs. Don't ask us to build a part of a name-value pair, because it's really hard to see when that's intended behaviour.

I don't really consider this a bug, more a sharp edge in the API.


Mmmm. So basically, i messed it up when put **?request=**


@GAZ082 the following should work:

``` py
import requests
import json

url = "http://server:8080/jsonrpc";
headers = {'content-type': 'application/json'};
payload = {'foo': 'bar'}
params = {'request': json.dumps(payload)}

r = requests.get(url, params=params, headers=headers)

print(r.url)
```

In the future though, questions like this will be answered quickly on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests)


Thank you sigma.
ImportError: cannot import name is_windows

I am not sure if this should be a bug for requests (2.5.2). httpie (http command) no longer works with requests 2.5.2. b/c the the method is_windows was removed from requests.compat

Traceback (most recent call last):
  File "/Users/csanchez/Projects/nti.dataserver-buildout/bin/http", line 413, in <module>
    import httpie.**main**
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/httpie-0.9.1-py2.7.egg/httpie/**main**.py", line 6, in <module>
    from .core import main
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/httpie-0.9.1-py2.7.egg/httpie/core.py", line 21, in <module>
    from httpie.compat import str, bytes, is_py3
  File "/Users/csanchez/Documents/workspace/nti.dataserver-buildout/eggs/httpie-0.9.1-py2.7.egg/httpie/compat.py", line 7, in <module>
    from requests.compat import is_windows, bytes, str, is_py3, is_py26
ImportError: cannot import name is_windows


This is a bug in httpie. We cleaned up old symbols we weren't using. `requests.compat` isn't a publicly documented or exported API.


I figured ..
Thanks a lot

Carlos


@papachoco  can you please share how you solved this error?


@ahsentaqikazmi this is an issue with httpie rather than Requests. You should be able to find solutions in the related issue for that project (jakubroztocil/httpie#314).
document combination of repeated response headers

as explained in https://github.com/kennethreitz/requests/issues/1155#issuecomment-53229867.

motivation:

requests uses a dict for response headers. In a given response, the same header may appear multiple times with different values, but dicts of course allow each key to map to only one value. requests [handles this intelligently](https://github.com/kennethreitz/requests/issues/1155#issuecomment-53229867), but users will not expect this given it's not documented in any of these places:
- http://docs.python-requests.org/en/latest/user/quickstart/#response-headers
- http://docs.python-requests.org/en/latest/api/?highlight=headers#requests.request
- http://docs.python-requests.org/en/latest/api/#requests.Response.headers

(Also, they may (likely) be unfamiliar with this part of http://tools.ietf.org/html/rfc7230#section-3.2, while only actually being familiar with werkzeug and other libraries that use a multidict instead to handle repeated headers.)

I think there's a perfect spot to document this behavior (right after the docs explain how the dict is already special in that it's case-insensitive), and given that #1155 will probably be closed, I figured it's a good time to document this and increase users' future understanding.

Thanks for the great work on requests.


:cake: Beautiful. =)


Thanks for merging. =)
multiple cookie headers should be joined with semicolons

please see https://github.com/kennethreitz/requests/issues/1155#issuecomment-55176733


This should be fixed by https://github.com/shazow/urllib3/pull/534 after we cut a new release of requests.


I apologise, my original statement in #1155 is wrong. Multiple cookie headers shouldn't be joined at all: there is simply no safe way to do that.
Update README to use Shields badges

The SVG badges served by [Shields](http://shields.io) are friendlier to retina displays and have a standardized design.

![Shields Comparison](https://cloud.githubusercontent.com/assets/2434728/6273906/0fc39ff8-b828-11e4-95e8-b7b5f80ef9e9.png)


No objection from me. @sigmavirus24?


One thing I noticed is that there's less precision in the download count – not sure if this problematic.


Meh. Download counts are working exactly correctly with PyPI at the moment anyway.


Thanks @iKevinY !


No worries!
[rfe] requests should process chunked responses transparently

When server responds with `Transfer-Encoding: chunked`, it would be nice if `requests` would process the chunked response. Right now if I do `response.iter_lines()` I get access to chunks part where length of the chunk is encoded with optional chunk extension. It would be awesome if `requests` would provide something like `iter_chunks` where I would get access to the chunk data only.

More info:
http://en.wikipedia.org/wiki/Chunked_transfer_encoding
http://mihai.ibanescu.net/chunked-encoding-and-python-requests


As mentioned in the blog post you link to, `httplib` does not provide us with access to those chunks. To get them we'd have to go around `httplib` and read directly from the socket itself. That's do-able, but it's a pretty aggressive layering violation. The first place to ask for that would be in our underlying layer, urllib3. I suggest opening an issue there. =)


Oh, should have read the blog most carefully.

Okay, will migrate this issue.


That's ok, it's an easy mistake to make. =)


@Lukasa Since it's in urllib3 now, how we can take advantage of it?


That's actually an interesting point.

By default, `iter_content` will automatically use `stream`, as you can see [here](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L656-L657). This _should_ mean that `iter_content` will now transparently work with chunked transfer encoding (interestingly, so will `iter_lines`).

However, it might be a good idea to document how best to handle chunked transfer encoding (namely, calling `iter_content` with a chunk size of `None`).


I probably won't have time to write that docs. But am really looking forward to try the new stuff in docker-py once urllib3 is rebased in requests and released.


See #2506 for those docs.