Raibek/requests-github-issues huggingface dataset:
gzip response is not decoded

It says in the FAQ:

> Requests automatically decompresses gzip-encoded responses, and does its best to decode response content to unicode when possible.

But the following code produces uncompressed data:

```
import requests
response = requests.get("http://www.shopcade.com/sitemaps/sitemap_products_index.xml.gz")
print response.text
```

So is this an issue of requests library or is there something wrong with the response from that site. When I download the archive via browser it decompresses well and the xml seems to be fine. Also the response headers contain the following key-value pair:

```
'content-type': 'application/x-gzip'
```


Ah, yes. This is an easy misunderstanding to make.

When we say gzip-encoded responses, we mean responses that are sent with `Transfer-Encoding: gzip`. That means that the body has a known type but has been compressed with gzip for transport. That does not include things whose actual content is gzip, as in the example above. Those are transmitted exactly as originally served.


Seems like a good thing to properly document for users so this confusion doesn't happen again.


@Lukasa:
I'm wondering if it's possible to use Requests to decode a gzip-encoded file (e.g. not-for-transport), or would that not be in the spirit of the library (file provenance and all)?

I've got a large number of gzipped text file to download then parse (that would be amazing to just stream), and if it was the server doing the encoding, it seems like Requests could handle it no problem, but if the file's already in that form, would it work?


@riordan we will not decode something that does not have a gzip (or compress) Content-Encoding. The gzip module in Python should do this for you though and it might be possible to create that with `response.raw` as a file object to sort of stream i.


Alternatively, you can set up a fairly simple generator-based pipeline:

``` python
import zlib

def decompress_stream(stream):
    o = zlib.decompressobj(16 + zlib.MAX_WBITS)

    for chunk in stream:
        yield o.decompress(chunk)

    yield o.flush()


r = requests.get(some_url, stream=True)
parseable_data = decompress_stream(r.iter_content(1024))
```


@sigmavirus24 @Lukasa This is _awesome_ thank you. That's exactly what I wound up doing.


#### my code:
```
def decompress_stream(stream):
    o = zlib.decompressobj(16 + zlib.MAX_WBITS)
    for chunk in stream:
        yield o.decompress(chunk)
    yield o.flush()

rsp = requests.post(url,stream=bUseStream, data=szCmd, verify=False, headers=loadCookie(url,headers),timeout=None)
       
    rsp.raise_for_status()
    f9 = open(szFNms1,"wb")
    for chunk in decompress_stream(rsp.iter_content(chunk_size=81920)): 
        if chunk: # filter out keep-alive new chunks
            f9.write(chunk)
            f9.flush()
            print("+",sep="",end="",flush=True)
    f9.close()
```
is error
```
('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))
```
@Lukasa
Allow a "url_prefix" to be set on Session

So we can do something like this:

``` python
COUCHDB_URL = 'https://mybigusername:myenourmouspassword@mycouchdb.provider.com/databasename/'

s = requests.Session(url_prefix=COUCHDB_URL)
r = s.get('/_all_docs', params=dict(include_docs=True))
r = s.get('/some_document_id')
```

The example uses the CouchDB API and it would almost entirely prevent the need for API wrappers like [this](https://github.com/adamlofts/couchdb-requests) (see it storing the url prefix [right here](https://github.com/adamlofts/couchdb-requests/blob/93e83d5e8cf7b0894de33747572789777dee845e/couchdbreq/resource.py#L80)). But extensive usage of any REST API would benefit from this.


Duplicate of #133.

In short, we're not adding new features to the API and we're especially not going to attempt to build URLs for you when you're using a session. There are lots of other libraries that allow for similar functionality but that is not something inherently necessary in the transmission or handling of HTTP. It's important to keep in mind that `requests` is "HTTP for Humans", not "HTTP and a bunch of super nice to have features for Humans". The main focus of this library is HTTP.


Well, thank you.
Upgrade urllib3 to 490d3a227fadb626cd54a240b9d0922f849914b4

This is the release tagged at 85dfc16817df1e3604c238ad5d64f3b229e0598b


Tested on OS X 10.10:

```
[tw-mbp-jsmith requests (urllib_upgrade)]$ make test
# This runs all of the tests. To run an individual test, run py.test with
# the -k flag, like "py.test -k test_path_is_not_double_encoded"
py.test test_requests.py
============================================================================== test session starts ===============================================================================
platform darwin -- Python 2.7.9 -- pytest-2.3.4
plugins: cov
collected 154 items 

test_requests.py ..........................................................................................................................................................

========================================================================== 154 passed in 26.34 seconds ===========================================================================
```

This is to get https://github.com/shazow/urllib3/pull/526 merged in https://github.com/pypa/pip/issues/2395 can take advantage of the fix for older SSL library versions. This was previously tracked at https://github.com/kennethreitz/requests/issues/2435.


We actually will pull in tip when we do a release. Speaking of which @Lukasa when do you think we should do a release? There's extra header awesomeness that's unreleased in urllib3 (merged last night) that I'd like to grab before doing a release.


@sigmavirus24 sounds good- I updated to HEAD, but feel free to drop this PR in favor of another one if this is in flight elsewhere.


It'd be great to get a bug-fix release soon (for issue #2435), so that we can use requests on python 2.7.9...


Howdy all, anything I can do to help get this over the finish line? Happy to pitch in on testing or updating docs for 'upgrading urllib3' if I missed something too.

Thanks!!


This is a pretty big blocker for me right now. Any chance we can get this pull request merged soon?


Thanks @sigmavirus24 !
Scaffold an API console sample from a model

Hello,

I wonder if RAML 
http://www.sitepoint.com/raml-restful-api-modeling-language/
http://raml.org/
couldn't be use to create (scaffold) an API console sample.

This idea is probably out of the scope of requests but we could talk about such idea here.

Same idea for server:
https://github.com/flask-restful/flask-restful/issues/394

Kind regards

PS: I'm writing this about RAML but maybe API Blueprint https://apiblueprint.org/ or Swagger http://swagger.io/ are also good candidates


> This idea is probably out of the scope of requests but we could talk about such idea here.

It is definitely out of the scope of requests. Since it is out of scope this is not the place to discuss it.
Raise explicitly when exclusive parameters are provided

It's a common case when one sends multipart request with json meta data. Currently, it is not clear that `json` argument is silently omitted. The same happens when both `data` and `json` are provided.


Thanks for the PR @andreif!

I'm mostly -0.5 on this. I think the right approach is to fix the documentation, since that's clearly what's lacking here (in fact, I could find no examples of using the json parameter actually, and I take full blame for that).

That said, the API for multipart requests seems confusing enough that I think some really good documentation around the `data`, `json`, and `files` parameters is really necessary.

Further, I think if we are going to raise an exception here, it should be a `ValueError`. `NotImplementedError` is usually used for abstract methods that need to be overridden by a subclass. That's not what we expect people to do here. What actually happens here, the problem is that we technically received to many values and can't possibly know what the users want us to do.


@sigmavirus24 Well, I wonder if we could take both files and json. It's a common case to upload files with json meta data, so maybe it makes sense to have a nice api for this.


It would be nice to write

``` py
r = requests.post('https://www.googleapis.com/upload/drive/v2/files',
                  params={'uploadType': 'multipart'},
                  headers={'Authorization': 'Bearer ' + access_token},
                  json={'title': 'test.txt'},
                  files={'file': open('test.txt', 'rb')})
```

instead of

``` py
r = requests.post('https://www.googleapis.com/upload/drive/v2/files',
                  params={'uploadType': 'multipart'},
                  headers={'Authorization': 'Bearer ' + access_token},
                  files={'file': open('test.txt', 'rb'),
                         None: (None, json.dumps({'title': 'test.txt'}),
                                  'application/json; charset=UTF-8')})
```


I could look into it if you are fine with accepting json argument together with files


@andreif That's a good idea, but sadly the API doesn't render out well. For instance, what happens if instead of uploading one file you're uploading two? And how do you associate them together?

Sadly, JSON in mutlipart/form-data messages is not standardised in any sense, so there's no way we can extend our API for that to be consistently useful. In that situation I'd want to resist the temptation to guess.

However, I'm +0 on throwing exceptions when mutually exclusive parameters are passed, because I don't really think we should fail silently. That said, it's backwards incompatible, and so we need to hold it for at least a minor release.


@Lukasa Alright. Should I change to `ValueError` in this case?


@andreif please do change it to a ValueError


@sigmavirus24 I took existing raise as an example. Should I change all of them to `ValueError` or should I keep the original as it is?

``` py
raise NotImplementedError('Streamed bodies and files are mutually exclusive.')
```


This should be solved with documentation.
Suppress the warning if verify=False

Setting verify to False differs from verify=None in being explicitly set,
and therefore the warning being undesirable.

See this for the a proposed alternative that involves importing
implementation details of Requests onto the application:
 https://github.com/kennethreitz/requests/issues/2214#issuecomment-72941896

This fix avoids deep imports and pulling the knowledge of internals of
Requests into the application.

Fixes #2214


Some notes:
1. urllib3 is a separate project that we vendor in its entirety. This PR affects it, and so requires a separate change in that library.
2. 'Explicit' is dependent on who you are. This is 'explicit' to the person who calls requests, but not to the person above them, if any. The warning is not just for the person writing against requests, but for anyone who uses an application that uses requests.

I'm -1 on this change.


`CERT_EXPLICITLY_NONE` means nothing and doesn't add much value here. Given that this is mostly in urllib3 and would rely on acceptance there, I'm closing this until progress has been made there.


Sorry, but the closing statement is false. CERT_EXPLICITLY_NONE means CERT_NONE set explicitly, and its value is making the distinction between such case and CERT_NONE set explicitly, purely by the lack of certificates.


@zaitcev The closing statement was: "Given that this is mostly in urllib3 and would rely on acceptance there, **I'm closing this until progress has been made there**." (Emphasis mine.)


@zaitcev `CERT_NONE`, `CERT_REQUIRED` and `CERT_OPTIONAL` are all defined on the `ssl` module and have a real meaning in the context of performing validation. `CERT_EXPLICITLY_NONE` means absolutely nothing in the same context.
Remove urllib3-specific section of iter_chunks

This commit resolves #2433 by using os.read() on self.raw.fileno()
in all cases, and removes the use fo the self.raw.stream() iterator
which would block rather than returning short reads.


I wonder if this shouldn't go into urllib3 instead, thereby giving the benefit there too. @shazow? @sigmavirus24?


I'd like that.


The existing documentation for `urllib3.response.stream` says:

> A generator wrapper for the read() method. A call will block until
> `amt` bytes have been read from the connection or until the
> connection is closed.

So from that perspective things are behaving as expected (it's supposed to block).  So...would the fix be a new method, or a boolean (e.g., `allow_short_reads`), or something else?


Hm, actually are we sure that doing something like .fileno() will work on AppEngine etc?


@shazow Maybe not?  Looking at the code in `urllib3.response`, `self._fp` is set from `body`, which could be a string type instead of a file pointer.

...oh wait, that's a lie, it couldn't be a string type in that case, but it only checks for a `read` method so it might not have `fileno()`, I guess.


Mmm, it's not clear that we can safely do this. A potential optimisation, though?


I'm open to have it use this method if it's of the right type, if that helps. :)


This breaks decoding of content too, no?
Implement limits for requests

Would anyone be interested in a `limit` parameter that timed out after X seconds or Y bytes received, as a DoS countermeasure? I want to implement one, but I'm not sure if you would accept the PR here or whether I should make it a third party library. It feels like it would be more fitting in requests itself, though.


We've been in a perpetual feature freeze for some time now (see https://github.com/kennethreitz/requests/issues/1165) meaning that this would not be accepted as yet another keyword parameter. Additions like this are being added to the [requests-toolbelt](/sigmavirus24/requests-toolbelt).


I'm proposing we close this since there's been no response in over 2 months and there seems to be no one else interested in this. Further, there's an appropriate place for a feature like this (the toolbelt).


Sorry, yes, do close it. I already made it into a library:

https://pypi.python.org/pypi/requests-guard
Upgrade/install of 2.5.0 fails on Fedora via yum

Running transaction check
Running transaction test
Transaction test succeeded
Running transaction (shutdown inhibited)
  Updating   : python-requests-2.5.0-3.fc21.noarch                                                                                                                                                             1/2 
Error unpacking rpm package python-requests-2.5.0-3.fc21.noarch
_error: unpacking of archive failed on file /usr/lib/python2.7/site-packages/requests/packages/urllib3: cpio: rename_
  Verifying  : python-requests-2.5.0-3.fc21.noarch                                                                                                                                                             1/2 
python-requests-2.3.0-3.fc21.noarch was supposed to be removed but is not!
  Verifying  : python-requests-2.3.0-3.fc21.noarch                                                                                                                                                             2/2 

Failed:
  python-requests.noarch 0:2.3.0-3.fc21                                                                    python-requests.noarch 0:2.5.0-3.fc21                                                                   


v 2.3 can be installed/uninstalled
v2.5 cann't be installed/upgraded-to
Works fine from pip


We do not package or build the Red Hat packages. Please take this up with downstream.


This seems to be caused by a conflict when the package is already installed via pip.
See https://bugzilla.redhat.com/show_bug.cgi?id=1177479#c4

One solution is to "pip uninstall requests". Or upgrade via pip instead of installing via rpm.
Update support documentation to be more accurate

We were missing instructions to report security vulnerabilities,
and all of the documentation referred to Kenneth as the only
source of support. We were also missing a link to StackOverflow.


One note. =D


Added you @Lukasa and added a bit more detail about vulnerability reporting.


This looks good.
- I might duplicate the vulnerability reporting text in a top-level page called "Security". Basically you want to make it really easy for anyone googling "python requests security" to find the page and if it's under a page heading called Support it's easier for Google/users to miss it
- I might break up the text into multiple paragraphs, as [people don't read on the Internet](http://www.nngroup.com/articles/how-users-read-on-the-web/), but they tend to fixate on the left edge of text (eg the beginnings of paragraphs).  


> I might duplicate the vulnerability reporting text in a top-level page called "Security". Basically you want to make it really easy for anyone googling "python requests security" to find the page and if it's under a page heading called Support it's easier for Google/users to miss it

"Security" is going to cause more search problems than not. I have a feeling people will be searching for that in relation to verifying/securing connections. I'm not adverse to this being a separate page (in fact I'd prefer it) but this will do for now.

> I might break up the text into multiple paragraphs, as people don't read on the Internet, but they tend to fixate on the left edge of text (eg the beginnings of paragraphs). 

Done


@Lukasa ping ;)


@sigmavirus24 Is it worth adding GPG key fingerprints for you and I?


Let me find mine, can you add yours as a comment?


Actually, I'll push mine, you can update yours since this is a branch on requests proper that we can both push to.


My key is there now. =)


Good to merge?


:+1:


:cake: :sparkles: :cake:
Docs fixed

Moved "Custom Headers" part after "More complicated POST requests" and before "POST a Multipart-Encoded File" in quickstart doc for adding headers is mentioned in the last one.


I think the correct fix is actually to come up with an example that doesn't require the `data` parameter. Custom headers should come before anything with 'more complicated' in the title. =)


I see, I just decided that it would be enough because custom headers are not mentioned in "More complicated POST requests".


Yeah, I can see that, but I think headers are the simpler topic. If you feel strongly about it though I can merge as-is and consider changing it later. =)


Moved it back and set example to use only headers kwarg without data. :)


Sorry for not being careful, now it finally seems like I fixed everything that was needed.


I think i prefer the existing form of this. Thank you, though!
Bug fix: field uri in digest authentication should not be empty when enc...

...ounter http redirections.

https://github.com/kennethreitz/requests/issues/2426


I haven't read this yet, and I'm certain it really is this simple, but I want to re read the RFC about this regardless to make sure this applies to all cases. I suppose we've never had trouble with this because people were using URIs that had paths when authenticating. It doesn't hurt to be safe though. =)

Thanks for your patience @luozhaoyu 


@sigmavirus24 Sure! It is rare because I could only reproduce it when use a proxy and encounter a redirect...


I'm comfortable with this. Sorry for the delay @luozhaoyu. I lost track of this.

Thoughts @Lukasa?


Fine by me. =)


Thanks @luozhaoyu 


@sigmavirus24 You're welcome!
Bug: field "uri" in header["Proxy-Authorization"] should not be empty

#### Issue

When I am handling [toolbelt's digest proxy issue](https://github.com/sigmavirus24/requests-toolbelt/issues/53), I find the proxy server rejects my request header:

Connection: keep-alive\r\n
Proxy-Authorization: Digest username="luozhaoyu", realm="balabalba@host.com", nonce="o3jFVAAAAADwDnHL3X8AABqmXw4AAAAA", uri="", 
response="9a5d356b4a5bac5473864896bcfdcc69", qop="auth", nc=00000001, cnonce="a681e67b2a3bb034"\r\n
Accept-Encoding: gzip, deflate\r\n

The problem is because field uri should be "/" rather than "". ([request-uri's definition is in RFC 2616](http://tools.ietf.org/html/rfc2616#section-5.1.2))
#### Bug reproduce

It is funny to find the bug. Set up a digest auth proxy, 
`r = requests.get(url, proxies=proxies, auth=auth, allow_redirects=True)`
url is http://bing.cn/ (this is a typo), there will be a HTTP 301 to http://www.rz.com/, client would try to access http://www.rz.com directly, but the proxy would reject and ask for user/password.
However, the automatic redirect's url parse result will be
`ParseResult(scheme='http', netloc='www.rz.com', path='', params='', query='', fragment='')` 
The path is empty, and will result in authentication failure.
#### Solution is one line:

add a line in `auth.py` line 110: `path = "/" if not path else path`
I will create a pull request later.


Closed by #2427.
Quoted newlines not handled properly

When trying to use iter_lines() reader for csv to read an parse a csv file it failed.  It turns out that if there is a quoted newline in the file it removes the comma quote delimiter for the quoted newline and the following line.

This makes it impossible to use iter_lines for csv.


## Can you use a paste servIce to give a concrete example of this? I'm not sure I follow your description 

Sent from my Android device with K-9 Mail. Please excuse my brevity.


Input file:
    "a","
    ","b"
    "c","
    ","d"

Output stings
    "a"
    "c"


I don't see how this is iter_lines' fault. It doesn't understand file formats, so it can't possibly know that it's in a CSV file and so that newlines can be quoted. It's not "iter_semantic_lines", and nor should it be. In this situation, you want to use iter_content.


Sorry, it looks like my fault.  I was faked out by my data source providing 2 different files with identical identifying headers.
Replace n-dash in HISTORY to workaround pip bug

There exists a problem (maybe a bug?) in pip when using a locale like `LC_ALL=C` (which is commonly used by CI environments and system configuration tools such as Puppet), where the PKG-INFO file is decoded using ascii, raising a UnicodeDecodeError when PKG-INFO contains non-ASCII characters (such as the n-dash removed by this commit):

```
Downloading/unpacking requests from https://pypi.python.org/packages/source/r/requests/requests-2.5.1.tar.gz
  Downloading requests-2.5.1.tar.gz (443Kb): 443Kb downloaded
  Running setup.py egg_info for package requests

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 104, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 245, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 1014, in prepare_files
    req_to_install.assert_source_matches_version()
  File "/usr/lib/python3/dist-packages/pip/req.py", line 359, in assert_source_matches_version
    version = self.installed_version
  File "/usr/lib/python3/dist-packages/pip/req.py", line 351, in installed_version
    return self.pkg_info()['version']
  File "/usr/lib/python3/dist-packages/pip/req.py", line 318, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/usr/lib/python3/dist-packages/pip/req.py", line 261, in egg_info_data
    data = fp.read()
  File "/usr/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 8161: ordinal not in range(128)
```

There are some existing issues for pip:
- pypa/pip#1233
- pypa/pip#761

Other projects have had the same issue occur; there is some useful discussion in these issues as well:
- cleder/fastkml#9
- gabrielfalcao/HTTPretty#108
- rbarrois/factory_boy#118

I know this isn't really a problem with requests (and should probably be addressed ultimately in pip), but I think replacing this character in HISTORY would be a pain-free and pragmatic way to help developers and system administrators, especially given the inactivity on the pip bug reports.

Many thanks!


Yeah, I'm just sick of this now. Thanks! :cake:


This is the second time these have snuck in. Where are they coming from? Do we think this closes #2419?


I hope so.
UnicodeDecodeError installing via pip-3.2

So here's a weird one. I have a Docker container based on debian:stable where I install python3 and python3-pip, and when trying to install requests from a requirements.txt file I get the following:

```
Downloading/unpacking requests (from -r /tmp/requirements/xxx.txt (line 2))
  Running setup.py egg_info for package requests

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 104, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 245, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/usr/lib/python3/dist-packages/pip/req.py", line 1014, in prepare_files
    req_to_install.assert_source_matches_version()
  File "/usr/lib/python3/dist-packages/pip/req.py", line 359, in assert_source_matches_version
    version = self.installed_version
  File "/usr/lib/python3/dist-packages/pip/req.py", line 351, in installed_version
    return self.pkg_info()['version']
  File "/usr/lib/python3/dist-packages/pip/req.py", line 318, in pkg_info
    data = self.egg_info_data('PKG-INFO')
  File "/usr/lib/python3/dist-packages/pip/req.py", line 261, in egg_info_data
    data = fp.read()
  File "/usr/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 8161: ordinal not in range(128)
```

However, when I run `pip-3.2 install requests` directly, it works fine.
I honestly have no idea where even to start on debugging this.


This is a duplicate of an old error that was fixed. Can you confirm that the version of requests is pinned to something in the 2.x line but is not 2.5.1?


#2196 is the already fixed issue for reference.


I've tried it with no version pinned, pinned to 2.5.1, and pinned to 2.5.0.
The thing that I don't understand is why it only fails when installing from a requirements.txt file.
Data argument introduced too early in Quickstart guide

I'm following the quick start guide in:
http://docs.python-requests.org/en/latest/user/quickstart/
I got confused when reading the section "Custom Headers" because the example code features the data argument, which hasn't been introduced in the guide yet (it is the next section "More complicated POST requests" that describes the data argument).
My suggestion is to swap the order of those two sections, i.e. have "More complicated POST requests" before "Custom Headers", or modify the example code in "Custom Headers" (remove data argument).


Agreed, this should be reworked. Marking as contributor friendly.


Why not just leave out the data argument in the Custom Headers section? That section is clearly not meant to explain the usage of data argument. It is being discussed later on in the "More complicated POST requests" section. What do you guys say?


The only reason to worry about it is that the header we're sending is a `Content-Type` one, which does rather require a body. 


Let's just set it to a useragent string then?


Totally reasonable. =)


Should I go ahead and submit a patch?

## Outline:
- Omit the data attribute.
- Replace the headers with a user-agent string


Make sure you change the verb, but sure. =)


The full user-agent string is too long:
`Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36`. Can I just write `Mozilla/5.0` or is it necessary to write the complete user-agent string?
Cookie header isn't passed when redirected

The `Cookie` header isn't sent to redirected URL's like other headers is:

```
# Requests version = 2.5.1

ipdb> response.request.headers
{'CustomHeader': 'this_still_works!'}
ipdb> response.history[0].request.headers
{'CustomHeader': 'this_still_works!', 'Cookie': 'token=top_secret_token; email="dev@dev.com";'}
```


There isn't enough information to even begin to debug this. You're clearly being redirected.
- Are you being redirected to new domain (or sub-domain)? (e.g., from `example.com` to `sub.example.com` or `example.org`)
- Are you being redirected from `http://` to `https://`?
- Where does the cookie come from?
  - Do you specify it yourself?
  - Is it in a cookie-jar to start with?
  - Are you using a session that collected the cookie for you?

These are just starting questions. I guarantee I'll have more once you answer them.


Apologies @sigmavirus24, definitely short on info:
- Redirect is append-slash like (`dev.local/api/test` -> `dev.local/api/test/`)
- Both served over HTTP
- Cookie is set in the headers dict passed to `requests.get`, no jar/session

Let me know if you need anything else.


So we [rebuild](https://github.com/kennethreitz/requests/blob/d2d576b6b1101e2871c82f63adf2c2b534c2dabc/requests/sessions.py#L168..L176) the `Cookie` header on a redirect as a matter of not exposing the cookie to people who shouldn't see it. Without a cookiejar to rebuild _from_ we won't persist the header through the redirect. We already [sanitize authorization](https://github.com/kennethreitz/requests/blob/d2d576b6b1101e2871c82f63adf2c2b534c2dabc/requests/sessions.py#L215) on redirect to different hosts, so we _could_ do the same here.

I want @Lukasa's thoughts on this though before I continue with work for this.

I, for one, think Cookie headers are in the same class as Content-Length and Host headers. They can be set by users but they really _shouldn't_ be set by users because it can do bad things if not handled with care.


Agreed, cookies set in the headers have an unclear scope, so we assume their scope is extremely tight. I think that's a good behaviour.


I agree, it's too much like "cookie injection" to send cookies as custom headers through redirects :)
Move noncebit to the only place it is used

Since we only allow for "auth" qop-value, hardcode it

Fixes #2408

I want to add tests but I'm not sure a good way to write them. =(


I merged the changes into my auth.py and it worked fine, both for my servers which offer qop="auth" and for those which offer qop="auth,auth-int".
From my point of view the issue #2408 is fixed by this commit.


Testing this is a nightmare, it's super stateful and you have to patch time and all sorts of weirdness. Let's just not.


I'm really considering spending a weekend or something adding tests around stuff and re-factoring it because we should be able to add tests for this kind of thing.
Environment http_proxy will cover the Session property s.proxies

When I use a proxy in my code I find a problem. In the code below

``` python
s = requests.Session()
s.proxies = {'http':'127.0.0.1:3128'}
r = s.get("http://www.apple.com")
```

If I set environmet var http_proxy='http://192.168.1.100:3128' then the request will use environment proxy rather than the proxy I have set in the code.
But if I use proxy like below the program will use the proxy which I set in the code.

``` python
s = requests.Session()
r = s.get("http://www.apple.com", proxies={'http':'127.0.0.1:3128'})
```

Is there any issue?


Possible duplicate of #2109


Nope, it's a definite duplicate of #2018.
Check that filenames are unicode or bytes

Instead of only checking one or another type of string-like object that we
accept, let's be able to check both.

Fixes #2411


Why are we not just using `basestring`?


@Lukasa This is what I get for working on this way too late last night.

@arthurdarcet thanks. Done.


Thanks! LGTM. :cake:


@Lukasa fwiw, I edited the commit message on my commit and retitled the PR to be more accurate descriptions of what's happening. The contents of the PR didn't change though.
urllib3 exceptions

Traceback (most recent call last):
  File "./dlcharts.py", line 111, in <module>
    content = get_url(url)
  File "/home/ed/projects/equalog/src/web.py", line 118, in get_url
    referrer=referrer)
  File "/home/ed/projects/equalog/src/web.py", line 81, in get_content
    content = response.content
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 694, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 627, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 242, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 200, in read
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='....', port=80): Read timed out.


What version of requests are you using? This should have been patched in 2.4.0.


I'm not sure what version it was. I was using the system package instead of my virtaulenv one, and that fixed it. Most likey diferent versions.
Wrong digest header creation when server has qop="auth,auth-int"

I always get "401 unauthorized" when trying to log in to a server which writes qop="auth,auth-int" in its answer. The problem doesn't exist with Curl.
I debugged the requests code and probably found the problem in auth.py. The function build_digest_header in class HTTPDigestAuth calculates
    noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, qop, HA2)
and gives back
    base += ', qop="auth", nc=%s, cnonce="%s"' % (ncvalue, cnonce)
to the server.

So in our client digest calculation we use qop="auth,auth-int", which we take from the server response, and make the server use qop="auth" in its calculation by giving back this header.

While debugging I can log in correctly if I change noncebit on the fly to
    noncebit = "%s:%s:%s:%s:%s" % (nonce, ncvalue, cnonce, "auth", HA2)
This would be my suggestion for the solution. Theqop for the calculation should be the same as we give back in our header.


We do not use `qop=auth-int`, we don't support it. When given multiple `qop` values like that in an auth challenge we're allowed to select one, and we have: `auth`.

You're right, however, about the fix. If we've chosen `auth` that's what we should use in our calculation. The simplest fix should be to pull the calculation of `noncebit` down into the only place it's used.

That said, can RFCreader @sigmavirus24 confirm that this is correct WRT the specs?


> That said, can RFCreader @sigmavirus24 confirm that this is correct WRT the specs?

Is this the new requests/urllib3 thing? If so, I thoroughly approve.


Thank you for your quick responses.
Would pulling down noncebit also fit for qop="auth,auth-int"? Wouldn't we still calculate with the wrong qop in case we match the condition "'auth' in qop.split(',')" as we would do with qop="auth,auth-int"?
I think we should not calculate with the qop we read from the response and instead with the one we write into the header (the constant "auth").


@berndschultze Sorry, I was insufficiently clear. I agree. What I meant was pulling that line into the `auth` block _and_ changing it to always specifically use `auth`.


@Lukasa You are right, I misunderstood. Thanks again.


I went back through [RFC 2617](https://tools.ietf.org/html/rfc2617#page-8) (which is still the definition of Digest Access Authentication, although parts were _updated_ (not obsoleted) by 7235). tl;dr the ABNF says unambiguously:

```
      qop-value         = "auth" | "auth-int" | token
```

Where `token` is one or more letters (`a-zA-Z`), digits (`0-9`), and US-ASCII visible characters (excluding `(),/:;<=>?@[\]{}` and `"`). In otherwords, _technically_ the server is doing the wrong thing but we can handle it because a `,` should never be part of the `qop-value` so it should be safe to approach this by following roughly this logic:

``` py
if 'auth' in set(qop_value.split(',')):
   qop = 'auth'
else:
   return None  # or whatever we return when we're not doing 'auth'
```


@berndschultze if you want to test out #2415 and let us know how that goes, that'd be awesome.
there is a bug when filename contains a non-ascii character

I ues python 2.7.9 and requests 2.5.1
when I do like this:

```
files = {'file1':('中文','hello')}
r = requests.post('http://test',files=files)
```

the request is like this(copy from fiddler):

```
Content-Disposition: form-data; name="file1"; filename*=utf-8''%E4%B8%AD%E6%96%87
```

I think the format of filename is wrong.

the right format should be:

```
Content-Disposition: form-data; name="file1"; filename="中文"
```


in requests\packages\urllib3\fields.py

```
def format_header_param(name, value):
    if not any(ch in value for ch in '"\\\r\n'):
        result = '%s="%s"' % (name, value)
        try:
            result.encode('ascii')
        except UnicodeEncodeError:
            pass
        else:
            return result
    if not six.PY3:  # Python 2:
        value = value.encode('utf-8')
    value = email.utils.encode_rfc2231(value, 'utf-8')
    value = '%s*=%s' % (name, value)
    return value
```

I think this is related code.

When the Content-Type is multipart/form-data,is it correct ues this function to format heaer?


Duplicate of #2313. Please search for existing issues before opening new ones in the future.
x509 Certificate 

Is there any support for x509 certificate authentication with requests? I can't seem to find any examples and would much prefer to use requests. 


By which you mean TLS authentication? Yes, it's enabled by default.


I apologise, I may have misread. Do you mean TLS _client_ certificates?


Yeah, sorry. I'm piecing together some horrible documentation from our vendor. The example app they gave is corrupted so awaiting that back. The language from documentation that I do have is

> In addition to using the certificate to establish the SSL connection, your system will need to use a client key and secret to authenticate to the APIs.


Ok, so we can use client certificates. You use the `cert` keyword as documented [here](http://docs.python-requests.org/en/latest/api/#requests.request).


Awesome. Thanks!
requests removes characters from valid url 

Requests strips the question mark when a hash tag follows. urlopen doesn't do it. Why?

Recreate bug with:

``` python
#!/usr/bin/python3

from urllib.request import urlopen
from requests import get


url = 'http://incolumitas.com?#param=value'

native = urlopen(url)
req = get(url)

assert native.url == req.url, '{} vs {}'.format(native.url, req.url)

"""
AssertionError: http://incolumitas.com?#param=value vs http://incolumitas.com/#param=value

Requests strips the question mark when a hash tag follows. urlopen doesn't do it. Why?
"""
```


The question mark indicates the beginning of the query portion of the URL. The octothorpe indicates the beginning of the fragment portion of the URL. We conclude that you have an empty query string and normalise it by removing the section entirely.

If the octothorpe is intended to be part of the query string you need to percent-encode it to avoid ambiguity. 

> On 10 Jan 2015, at 12:52, NikolaiT notifications@github.com wrote:
> 
> Requests strips the question mark when a hash tag follows. urlopen doesn't do it. Why?
> 
> Recreate bug with:
> 
> #!/usr/bin/python3
> 
> from urllib.request import urlopen
> from requests import get
> 
> url = 'http://incolumitas.com?#param=value'
> 
> native = urlopen(url)
> req = get(url)
> 
> assert native.url == req.url, '{} vs {}'.format(native.url, req.url)
> 
> """
> AssertionError: http://incolumitas.com?#param=value vs http://incolumitas.com/#param=value
> 
> Requests strips the question mark when a hash tag follows. urlopen doesn't do it. Why?
> """
> —
> Reply to this email directly or view it on GitHub.


I understand. Many thanks for the quick response. Will close.


@NikolaiT just to give you some more detail about this:

The URI structure (that URLs follow) is of the form `{scheme}://{authority}{/path}{?query}{#fragment}`. (On Python 3) The `urllib.parse` module gives us the `urlparse` function to look at these components of a URL.

``` py
>>> import urllib.parse
>>> uri = urllib.parse.urlparse('http://incolumitas.com?#param=value')
>>> uri
ParseResult(scheme='http', netloc='incolumitas.com', path='', params='', query='', fragment='param=value')
>>> uri.geturl()
'http://incolumitas.com#param=value'
```

According to this module (and the RFC that defines handling of URIs) these two URLs are equivalent. This can be seen by visiting both in your browser. In essence, if we didn't modify the URL it would be as valid as our current approach is. We do normalize URLs though because servers (and sometimes users) give us some rather bizarre URLs that will only "just work" when normalized. As core developers of a library whose core design goal is to make users' lives better, we need to take this approach to satisfy that goal.

I hope that helps give you a deeper understanding of what's happening, why it is okay, and why it _should_ happen.
Allow a 'verbose' flag for raise_for_status()

raise_for_status() is super awesome, but a lot of the time, the error code and message is not informative enough to assist with debgugging when an exception is raised. It would be nice to have a verbose flag for raise_for_status that would also print the response body, where more details are often included. I would be happy to work on this, but wanted to make sure it is not already in the works.


If you do:

``` py
import requests

r = requests.get(url)

try:
    r.raise_for_status()
except requests.exceptions.HTTPError as error:
    print(error)
    print(error.response.text)
```

We don't print the body (and likely won't add it, even as a flag) for several reasons:
- It may not exist (and therefore wouldn't be helpful)
- It could be plain binary data which will certainly not help anyone
- It could have characters that cannot be printed (and will cause even more problems)

You can certainly do this yourself and based on some searches on StackOverflow and our issues, it doesn't seem like anyone has had the need for this before.

I'm -0.5 if only because `verbose` would obviously default to `False`.


Thanks for the quick response @sigmavirus24. That makes a lot of sense. I was thinking that a verbose flag would avoid the need for the try/except structure, but it is simple enough to a make a little helper function that processes the request and does that. Thanks again! Please go ahead and close this out.


Verbose flag would be very nice
Clarify InsecureRequestWarning and verify=True in docs

The docs:

http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification

say:

> By default, verify is set to True. Option verify only applies to host certs.

If my HTTPS connection is being verified by default, why am I also seeing a InsecureRequestWarning?  Clarifying this situation and updating the docs with more info would be helpful IMO.


> If my HTTPS connection is being verified by default, why am I also seeing a InsecureRequestWarning? Clarifying this situation and updating the docs with more info would be helpful IMO.

I don't know why. Can you give us any details to help you?


> I don't know why. Can you give us any details to help you?

Thank you for the prompt response.  Actually, this was my fault.  I didn't realize I was using a session that had verify=False.  The warning did exactly what it was supposed to, I just thought it was someone elses's fault!  :o/

Thanks again.


Hah. No worries.
how  can I get the "exception code"

I am using "requests (2.5.1)" .now  I want to catch the exception and return an dict with some exception message,the dict I will return is as following:

```
{
   "status_code":   61, # exception code,
   "msg": "error msg",
}
```

but now I can't get the error status_code and error message,I try to use 

```
except requests.exceptions.ConnectionError as e:
response={
            u'status_code':4040,
            u'errno': e.errno,
            u'message': (e.message.reason),
            u'strerror': e.strerror,
            u'response':e.response,
        }
```

but it's too redundancy,how can I get the error message simplicity?anyone can give some idea?


Questions belong on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).
verify is sticky in Session

I'm using Session.request to do GET from a HTTPS URL where the the server side certificate may be invalid.  For testing purposes, if the certificate is invalid, I want to retry the GET with 'verify=False'.  I'm reusing the Session for the retry, but the second request also fails even though I specify 'verify=False' in the request.  It seems that the verify state is persisted in the Session somewhere and not updated from the request parameters if already set in a previous request.  Is this the expected behavior?  It seems unusual to me.  If I create a new Session for the second request the certificate doesn't get checked in this request. 


This is a duplicate of #2255.
Attempt to quote anyway if unquoting fails

Fixes #2356

TODO
- [x] Add test


Looks fine, modulo a test. =)


where does the list of safe characters come from? maybe this could be pulled into a named var for clarity?


## It comes from the RFCs

Sent from my Android device with K-9 Mail. Please excuse my brevity.


Update?


Update: forgotten. I've been really busy lately but I'll prioritize this. Sorry for losing track of it.


Added two tests and gave the strings slightly better names. Not sure we need to spell out every last RFC referenced in the comments though since they're all pretty well known. @Lukasa I'd appreciate some review.


Some small notes. =)


@Lukasa forgot to mention this was updated to address your feedback.


LGTM. :+1:
Using python-nss for SSL

For HTTPS urls where Ceritifcate verification is required, I would like to use certificates/keys stored in NSS database(Mozilla NSS), There is python-nss module which provides the interface to access the NSS Database. 

Is there a way i can use requests module to use python-nss. Any hints on how to achieve this would be helpful. 


Currently, requests cannot use python-nss directly. Assuming all you want to do is use the certificates and keys, you'll need to transform them into a `.pem` file that OpenSSL can read. That's extremely painful, I'm afraid, as OpenSSL does not let you pass it `.pem` files in memory.

Unfortunately, enhancing requests to do this is extremely tricky and really needs to be done lower down the stack. At this time, there is no plan to do that work: certainly not in the requests project.

I'm sorry I can't be more helpful.