Raibek/requests-github-issues huggingface dataset:
invalid charset

r=requests.get("http://export.yandex.ru/weather-ng/forecasts/26686.xml")
print r.encoding

> > ISO-8859-1
> > print r.text.**class**
> > <type 'unicode'>

cant write to file valid content

---

urllib return <type 'str'>, read/write to file dont break text


You could call `.encode()` on `r.text`. If there aren't any characters 
that can be converted to ASCII, try `.encode('utf-8')` and that should write 
without a problem to your file, you'll just need to open the file using 
`codecs.open`.


perhaps we should fallback to chardet detection if the provided encoding doesn't exist on the system.


``` pycon
>>> codecs.lookup('utf-8')
<codecs.CodecInfo object for encoding utf-8 at 0x100371b48>
>>> codecs.lookup('utf-fake')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown encoding: utf-89
```


Should be fairly simple.


Why was this closed?


Not sure why it was closed, but charade has problems with this. Re-ticketing that portion over there sigmavirus24/charade#2.


Excepting the charade problems which are ticketed there, this is resolved. =)
requests.post() adds unwanted Authorization: Basic header with anonymous:no@mail.dom

```
import requests
requests.post("http://localhost:9091/", data="foobar")

$ nc -l -p 9091
POST / HTTP/1.1
Host: localhost:9091
Content-Length: 6
Authorization: Basic YW5vbnltb3VzOm5vQG1haWwuZG9t
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/0.14.2 CPython/2.7.3 Linux/3.6.0-1-ARCH

foobar^C

$ base64 -d
YW5vbnltb3VzOm5vQG1haWwuZG9t
anonymous:no@mail.dom
```

WTF this is coming from? So after poking a bit, i found that it's from my .netrc: `default login anonymous password no@mail.dom`

Okay, so i have to disable automatic netrc support somehow.

Please, tell me one thing, how does it correspond to #44 where auth is explicitly set, but no header is sent and here no auth is set but header is sent?


Interesting. I'd remove that from your `.netrc` immediately, fwiw :)

Also, there's a config you can set to diasble this: `trust_env=False`


Thank you. I did not touch .netrc in years, i think this line was useful for browsing public FTP.

Still it is not clear why in one case requests does not send explicitly set auth headers on premise that exact algorithm (Basic, Digest, etc) is not known beforehand. Does .netrc somehow imply Basic authorization?


If you think about the Requests API this actually makes sense. Requests special-cases Basic Auth so you can just pass a naked tuple to the `auth` parameter, but otherwise you need to pass a class that implements a few special methods. There's no provision for you to pass anything to the `auth` parameter that means "Use my .netrc file with this authentication type". Maybe there should be, but right now, if you haven't told Requests what auth type to use, it will assume you mean Basic Auth, whether you use a tuple or .netrc.

Does that answer your question?


@Lukasa there shouldn't be such a parameter ;)


You shouldn't have that line in your `.netrc`. If it's extremely common, we can consider special-casing it.


@Lukasa Yes, it does, thank you for thorough answer. And i just tested, the #44 would be satisfied now that requests actually send Basic right away. Everything is consistent now.

Thank you.
Installation problems on Windows

I wanted to install using pip and it gives me the following error:

``` python
PS C:\Users\bla> pip install requests
Downloading/unpacking requests
  Real name of requirement requests is requests
  Downloading requests-0.14.2.tar.gz (361Kb): 361Kb downloaded
  Running setup.py egg_info for package requests
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "C:\Users\patrick\build\requests\setup.py", line 6, in <module>
        import requests
      File "requests\__init__.py", line 52, in <module>
        from . import utils
      File "requests\utils.py", line 22, in <module>
        from .compat import parse_http_list as _parse_list_header
      File "requests\compat.py", line 112, in <module>
        from .packages import chardet2 as chardet
    ImportError: cannot import name chardet2
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "C:\Users\patrick\build\requests\setup.py", line 6, in <module>

    import requests

  File "requests\__init__.py", line 52, in <module>

    from . import utils

  File "requests\utils.py", line 22, in <module>

    from .compat import parse_http_list as _parse_list_header

  File "requests\compat.py", line 112, in <module>

    from .packages import chardet2 as chardet

ImportError: cannot import name chardet2

----------------------------------------
Command python setup.py egg_info failed with error code 1
```

I can import chardet as chardet from the interpreter. importing chardet2 fails.
i guess this is a dependency that wasn't properly updated.
the python version is 3.2.2


Have you searched for `chardet2` in opened issues before raising this? It's a duplicate.


i searched for windows. sorry
cookies problem

Hi, Iam trying to post a form with cookies.
When I post the form and see the page with the results, there is an error in the page, saying that I didnt fill one field right. Iam absolute sure I fill that field right. When I post the data with web browser, its all right.
I think it must be cookie problem, becose when I disable the cookies in the browser and post the right data, it shows the exact same error when I send it via Python.

There is the code:

``` python
# -*- coding: utf-8 -*-
import urllib, urllib2
import cookielib
import requests
adress= 'http://....'
param= {
    'name' : '....',
    'email' : '....',
    'phone' : '....',
    'message' : '....'}#etc... 
headers = {
      'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1',
      'Connection': 'keep-alive',
      'Accept-Language': 'en-us,en;q=0.5',
      'Accept-Encoding': 'gzip, deflate',
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' }
c=requests.session()
requestp = c.get(adress)
SSID=requestp.cookies['PHPSESSID']
cookiesp = dict(PHPSESSID=SSID)
request = c.post(adress, data=param, headers=headers)
print request.headers          #headers of the result webpage
print request.request.headers  #headers posted
```

Output:

```
{'content-length': '6911', 'x-powered-by': 'PHP/5.3.17-1~dotdeb.0', 'expires': 'Thu, 19 Nov 1981 08:52:00 GMT', 'server': 'nginx/1.2.4', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'max-age=200', 'date': 'Thu, 01 Nov 2012 19:44:14 GMT', 'content-type': 'application/xhtml+xml; charset=utf-8'}
{'Accept-Language': 'en-us,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1', 'Cookie': 'PHPSESSID=2des6ks39kouoqvn3u6l425kp6', 'Content-Type': 'application/x-www-form-urlencoded'}
```

If you have same ideas what is wrong, I would so gratefull. Thank you very much and I apologise for my english (:

Ok, I solved this with mechanize... can you delete it?


Resolved by user intervention.
raise_for_status should raise when status_code is None

r.raise_for_status and r.ok should not report success when response has wonky data. this occurs when you use grequests and the greenlet encounters problems (timeout, etc)

``` python
import grequests
urls = [grequests.get('http://google.com/?q={}'.format(n), timeout=.001) for n in range(3)]
responses = grequests.map(urls, size=2)
print '# responses={}'.format(len(responses))
for r in responses:
    print 'status_code={}; ok={}; error={}'.format(r.status_code, r.ok, r.error)
    r.raise_for_status()
```

```
Traceback (most recent call last):
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/requests/models.py", line 631, in send
    raise ConnectionError(sockerr)
ConnectionError: [Errno 65] No route to host
<Greenlet at 0x1013f2c30: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

Traceback (most recent call last):
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/requests/models.py", line 631, in send
    raise ConnectionError(sockerr)
ConnectionError: [Errno 65] No route to host
<Greenlet at 0x1013f2af0: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

Traceback (most recent call last):
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/darrell/.virtualenvs/clarence/lib/python2.7/site-packages/requests/models.py", line 631, in send
    raise ConnectionError(sockerr)
ConnectionError: [Errno 65] No route to host
<Greenlet at 0x1013f27d0: <bound method Request.send of <Request [GET]>>(prefetch=True)> failed with ConnectionError

# responses=3
status_code=None; ok=True; error=None
status_code=None; ok=True; error=None
status_code=None; ok=True; error=None
```


You might want to rebase this pull request to get Travis to test it again. There's really no reason for it to have failed. =)


Close/Repopen to trigger rebuild.


May have to update some of the tests as well ;)


Closing for now. Reopen if you update!
Added 'zip_safe' flag to 'setup.py' & Fixed typo in 'MANIFEST.in'

While installing `requests` using the `python setup.py --quiet install` command, I noticed the following warnings:

```
warning: no files found matching 'tests/*.'
zip_safe flag not set; analyzing archive contents...
requests.certs: module references __file__
```

After a bit of digging I found out that the first of those had to do with the `tests/*.` (notice the dot) pattern used in `MANIFEST.in`, so that I corrected it in the second of the associated commits. The other two messages (the last of which is the reasoning behind the choice) show that the package is not ["zip-safe"](http://bit.ly/PjdL9X), so that I added the `zip_safe=False` flag to the `setup.py` file in the first of the associated commits.


thanks!
Settings wrong form-data format for params when used with file upload

Hey to all, and thank you for great library!

I might have found an issue with the way of requests handle file POST upload when they are sent with additional params which aren't files. It adds an extra `Content-Type: text/plain` to fields that aren't a file. (according to the W3c and from sniffing my own browsers to be sure)

For example, when I run this :

```
requests.post('localhost',files={'file1.txt' : open('file1.txt','rb')},data={'submit-name':'Larry'})
```

or this (I don't really which one to prefer I guess `data` over `params`) : 

```
requests.post('localhost',files={'file1.txt' : open('file1.txt','rb')},params={'submit-name':'Larry'}))
```

requests send this : 

```
   Content-Type: multipart/form-data; boundary=AaB03x

   --AaB03x
   Content-Disposition: form-data; name="submit-name"
   Content-Type: text/plain

   Larry
   --AaB03x
   Content-Disposition: form-data; name="files"; filename="file1.txt"
   Content-Type: text/plain

   ... contents of file1.txt ...
   --AaB03x--
```

when it should have sent this :

```
   Content-Type: multipart/form-data; boundary=AaB03x

   --AaB03x
   Content-Disposition: form-data; name="submit-name"

   Larry
   --AaB03x
   Content-Disposition: form-data; name="files"; filename="file1.txt"
   Content-Type: text/plain

   ... contents of file1.txt ...
   --AaB03x--
```

I think the issue is more urllib3 related than requests related and I've opened an issue here (https://github.com/shazow/urllib3/issues/111) but maybe I am wrong or maybe you guys would want to know!


Looks like your `urllib3` pull request was accepted, so I guess we can consider this resolved.

Useful note: `params` does not do what you think it does. It builds a query string:

``` python
>>> r = requests.get('http://www.google.co.uk/', params={'key': 'value'})
>>> r.url
u'http://www.google.co.uk/?key=value'
```
Requests 0.14.2 fails with Python 3.2 on Travis.ci

Somehow chardet2 is missing;

```
File "requests/compat.py", line 112, in <module>
from .packages import chardet2 as chardet
ImportError: cannot import name chardet2
```

See this log output for details: https://travis-ci.org/#!/zynga/jasy/builds/2972117


0.14.1 works fine BTW


Duplicate issue [here](https://github.com/kennethreitz/requests/issues/916).


Closed: duplicate.
oauthlib should not use random.getrandbits()

The function generate_nonce() of the module requests.packages.oauthlib.common uses random.getrandbits() to create the nounce. The global mersenne twister PRNG instance should not be used for the task. For example the instance isn't reseeded on fork() which results in a very high probability of collisions in a forking server environment. Python's tempfile module contains an implementation that avoid the issue.

I suggest that you use os.urandom() for the job. It's available on all major platforms. The platform's CPRNG makes it inherently secure, too.


Wrong project. You should open this on idan/oauthlib
Tests missing in requests-0.14.2.tar.gz

`tests` directory is missing in requests-0.14.2.tar.gz (available on PyPI). It was present in requests-0.14.1.tar.gz.


There are a few problems associated with the packaging of 0.14.2, as you can see [here](https://github.com/kennethreitz/requests/issues/916). These are being resolved as part of #939. For the moment I'm closing this, but if the next patch release of `requests` is still affected, please reopen this issue.
Fix IRI encoding issue [updated]

We ran in to this issue where IRIs convert to URIs (through Django's iri_to_uri) would blow up in requests.

The issues seems to be when IRIs are converted to URIs they are percent-quote encoded, and when requests processes it with IDNA, it blows up because the URL components exceed 64 characters (since now it's three characters per actual character at least). However, they are valid URLs because they may encode to fewer than 64 characters in IDNA.

The solution is to reverse the percent-quote encoding, and then re-encode with IDNA.

Update: Solved merge conflict


Would the `encode_urls` configuration assist you here?


@kennethreitz I don't see `encode_urls` anywhere.


What `encode_urls` are you referring to? Is there a better around this?
Make sure that url has fragment removed when using a proxy

When making a request that uses a proxy Request.path_url falls back to Request.full_url. However Request.full_url does not strip the fragment from the url, which is improper and will cause some HTTP servers to respond with a 400 error.

This patch modifies Request.path_url to pass full_url through urlparse.urldefrag before returning it when a proxy is being used. It also plumbs urldefrag through the compat module so that I can use it here.

The alternative would be to modify full_url to defrag, but it seems to be used in setting the url for the Response object so I figured it would be better done in here.


Looks good, thanks!
Fix IRI encoding issue

We ran in to this issue where IRIs convert to URIs (through Django's `iri_to_uri`) would blow up in requests.

The issues seems to be when IRIs are converted to URIs they are percent-quote encoded, and when requests processes it with IDNA, it blows up because the URL components exceed 64 characters (since now it's three characters per actual character at least). However, they are valid URLs because they may encode to fewer than 64 characters in IDNA.

The solution is to reverse the percent-quote encoding, and then re-encode with IDNA.


(Rebased)


Can you send another pull request? Merge conflict.
FR: Include round trip time as a response attribute

This is a feature request.

It would be very useful if the requests package included a round trip time attribute on responses received.  pyCurl includes this functionality via pycurl.TOTAL_TIME.  

This can be worked around currently with something like this:

``` python
gc.disable()
start = time.time()
request = requests.get('https://github.com/timeline.json')
total = time.time() - start
gc.enable()
request.total_time = total
```

However, it would be nice to have this attribute generally available.  I am not sure if the garbage collection disable/enable is really necessary or not.


This has been considered in the past, but I've turned it down because it's simply out of scope.

It's quite simple to do this yourself with hooks, however.


@bmannix
See issue #361.

@kennethreitz
How about contrib module?


No, but thank you for the suggestion.

Sent from my iPhone

On Oct 24, 2012, at 1:26 PM, Piotr Dobrogost notifications@github.com wrote:

> How about contrib module?
> 
> —
> Reply to this email directly or view it on GitHub.


Yup.  Hooks look like a very good solution for this problem.  The more I learn about requests, the more I like it.  Thanks for your work!


They are good as long as you don't want to have specific [timings](http://curl.haxx.se/libcurl/c/curl_easy_getinfo.html) which curl provides. These need to be measured at the socket level at which requests do not operate directly.
Remove path hacks that break chardet in Python 3. Closes #858.

This replaces the sys.path hack with a slightly less objectionable
sys.modules hack.

Both have the effect of making the vendored lib's absolue imports work
as expected when oauthlib isn't installed. The sys.modules hack doesn't
insert the rest of the vendored crap in a global path, however.


Wow, I like this a lot better actually. Thanks!


FYI, this sparked some very interesting conversations at PyCarolinas :)
issue-901

RFC 2616 says that header values should be case insensitive. This patch fixes Content-Encoding to be case insensitive. This particular header was giving me grief - there may be others that need the same treatment.

Pull request has been submitted.


Changes to urllib3 should be submitted there, not here. Once that's accepted 
you can make the pull here for your changes to utils unless they're 
independent of each other.
On Fri, Oct 19, 2012 at 11:56:22AM -0700, Michael wrote:

> RFC 2616 says that header values should be case insensitive. This patch fixes Content-Encoding to be case insensitive. This particular header was giving me grief - there may be others that need the same treatment.
> 
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/building39/requests issue-901
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/kennethreitz/requests/pull/902
> 
> -- Commit Summary --
> - issue-901
> 
> -- File Changes --
> 
> M requests/packages/urllib3/response.py (2)
> M requests/utils.py (5)
> 
> -- Patch Links --
> 
> https://github.com/kennethreitz/requests/pull/902.patch
> https://github.com/kennethreitz/requests/pull/902.diff
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/902


OK, cool. I'll get that done.

On 10/19/2012 02:00 PM, Ian Cordasco wrote:

> Changes to urllib3 should be submitted there, not here. Once that's 
> accepted
> you can make the pull here for your changes to utils unless they're
> independent of each other.
> On Fri, Oct 19, 2012 at 11:56:22AM -0700, Michael wrote:
> 
> > RFC 2616 says that header values should be case insensitive. This 
> > patch fixes Content-Encoding to be case insensitive. This particular 
> > header was giving me grief - there may be others that need the same 
> > treatment.
> > 
> > You can merge this Pull Request by running:
> > 
> > git pull https://github.com/building39/requests issue-901
> > 
> > Or you can view, comment on it, or merge it online at:
> > 
> > https://github.com/kennethreitz/requests/pull/902
> > 
> > -- Commit Summary --
> > - issue-901
> > 
> > -- File Changes --
> > 
> > M requests/packages/urllib3/response.py (2)
> > M requests/utils.py (5)
> > 
> > -- Patch Links --
> > 
> > https://github.com/kennethreitz/requests/pull/902.patch
> > https://github.com/kennethreitz/requests/pull/902.diff
> > 
> > ---
> > 
> > Reply to this email directly or view it on GitHub:
> > https://github.com/kennethreitz/requests/pull/902
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/kennethreitz/requests/pull/902#issuecomment-9613705.


Closing, and will resubmit without the urllib3 change.


Sounds good.
Content-Encoding expects values to always be lower case

RFC 2616 says that header values should be case insensitive. This patch fixes Content-Encoding to be case insensitive. This particular header was giving me grief - there may be others that need the same treatment.


Was this supposed to be a pull request?


I just now submitted the pull request.

On 10/19/2012 01:49 PM, Ian Cordasco wrote:

> Was this supposed to be a pull request?
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/kennethreitz/requests/issues/901#issuecomment-9612456. 


Resolved in #903.
Added ability to choose SSL version

This also acts as a workaround for #827, which in some cases, if you are
connecting to an SSL server that does not support SSLv2 or SSLv3 (and,
for exapmle, only supports TLSv1), would cause an illegal EOL SSL error.
If you select the right SSL protocol version, you won't see this error.

Kenneth, it looks like you tried to fix this in a previous revision by hard-coding the SSL version, and then you rolled it back.  This makes it configurable via parameter to requests.requesttype()


I made one last change to correct an issue where you could have a crash if Python was built without SSL support.

Also, I've submitted all of these changes to the urllib3 project, and it looks like @shazow is interested in merging, so you should be good there.


Interesting - build 802 failed because it timed out after 15 minutes. 803 succeeded, but it finished before 802.  So, this pull request is being reported as a failure when it shouldn't be...


I won't be merging this at this time. Requests should be able to automatically fall back, just like the web browser.

I'll be adding TransportAdapters soon, and you'll be able to specify advanced options like this.


That's the problem.  In my environment, on Windows (but not Linux), when I'm connecting to to an HTTPS server that only supports TLSv1/2/3, but NOT SSLv2/3, I get the same error that's reported in #827:

`requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol`

That's why I submitted this as a workaround.


I'll be adding TransportAdapters soon, and you'll be able to specify advanced options like this.
Transport Adapters

Let's not wait for urllib3 for this.

http://kennethreitz.com/the-future-of-python-http.html
- Requests would ship with HTTP and HTTPS TransportAdapters.
- Take the current urllib3 integration and move it into said adapters.

As far as I can tell, we'd need two types of Adapters:
- Connections
- Requests/Responses

Perhaps we could just have a connection adapter, which is akin to a Requests' Session.


Can you point towards the current urllib3 integration?


Everything in Request.send :)


Hmm, I think the example in the post may work well:

```
class DistributedAdapter(BaseAdapter):
    def __init__(self):
        self.connect_pool = …

    def send(self, request):
        """Takes a Request object, returns a Response object."""
        # Whatever needs to happen here.
        …

```


Ah thanks.


Started branch 'adapters'


Converted to a pull request.


Let's add a new attribute to sessions, `adapters`.

Also a new method, `Session.mount`:

``` python
s = requests.session()
s.mount('null:', DevNullAdapter())
s.mount('http://haystack', WsgiAdapter(app=haystack))
```


:D


``` pycon
>>> s = requests.session()
>>> s.adapters
{'http://': <HTTPAdapter>, 'https//': <HTTPSAdapter>}

# Register a new adapter.
>>> s.mount('http://haystack', FakeAdapter(app=haystack))
>>> s.adapters
{'http://': <HTTPAdapter>, 'https://': <HTTPSAdapter>, 'http://haystack': <FakeAdapter>}

```


Ad-hoc request logic:

```
s = requests.session()
s.get('http://...')
# request = Request(...)
# s.adapters['http://'].send(request, prefetch=True)
```


Need to figure out config propagation. 


The release of this may be v1.0.0, possibly.


Going to refactor a bunch of stuff for this. Simplify all the things. Remove ad-hoc configuration, verbose mode (use real logging), session arguments, etc.


Replace `prefetch` with `stream` everywhere.


@kennethreitz don't forget to reopen this one too ;)


development will continue in the master branch.


#969
Add EditorConfig file

This adds an [EditorConfig file](http://editorconfig.org) to denote code style for `*.py` and `*.rst` files.  This EditorConfig file is a human and machine readable code style guide.  Currently EditorConfig files can be read by various text editors/IDEs using plugins.

This EditorConfig file correctly reflects your code style except for the following errors which were revealed by an EditorConfig checker [I'm developing](https://github.com/treyhunner/editorconfig-tools):

```
docs/community/support.rst: No final newline found
docs/dev/authors.rst: No final newline found
docs/_themes/README.rst: Tab indentation found
docs/conf.py: No final newline found
requests/packages/oauthlib/common.py: Trailing whitespace found
requests/packages/oauthlib/oauth1/rfc5849/signature.py: Trailing whitespace found
```

All of the above errors except for the last two (in `requests/packages`) would be resolved by #892.


Hmm, I think this is a bit out of scope for now. I'll definitely consider it in the future though. Trying to keep things minimal :)

Thanks!
error when trying to use oauth

So I've been porting my code from using requests-oauth to the built in requests oauth. So far I haven't had any luck. Here is what I'm doing:

https://github.com/shelfworthy/flixpy/blob/requests_oauth/flixpy/client.py#L41

vs. before with requests-oauth:

https://github.com/shelfworthy/flixpy/blob/master/flixpy/client.py#L20

any guidance?


I've spent some more time on this and think I know what is causing the error. The url I get from requests looks like this:

```
http://api-public.netflix.com/catalog/titles?term=thor&oauth_nonce=38117121406491052691350135419&oauth_timestamp=1350135419&oauth_consumer_key=k3cuh3wwvdfe434rc8pzyng&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&v=2.0&output=json&oauth_token=None&oauth_signature=yTNhyB%2FtfMNUa3bGiWWPHtcnPjQ%3D
```

vs the one I get with my old method:

```
http://api-public.netflix.com/catalog/titles?term=thor&oauth_nonce=30581223&oauth_timestamp=1350135535&oauth_consumer_key=k3cuh3wwvdfe434rc8pzyng&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&v=2.0&output=json&oauth_signature=0hx3wJoc4235hPHvKEFz63x5%2Bs%3D
```

I'm guessing the issue is the `oauth_token=None` in the requests version.


looks like the issue here is that I need to convert all strings to unicode before sending them in. I need to add some logic so that the values that are `None` don't get converted to `u'None'` and in theory I should be good.


ok, so this did come down to unicode. Once everything is unicode encoded stuff works without issue.


Yeah, unfortunately we are sensitive to encoding, so our two options were
to require you specify an encoding, or require Unicode for all parameters.
We went with the latter, sorry for lack of docs. :(

On Oct 13, 2012, at 16:02, Chris Drackett notifications@github.com wrote:

looks like the issue here is that I need to convert all strings to unicode
before sending them in. I need to add some logic so that the values that
are None don't get converted to u'None' and in theory I should be good.

—
Reply to this email directly or view it on
GitHubhttps://github.com/kennethreitz/requests/issues/891#issuecomment-9404391.


why not convert to unicode in requests, just to make sure?


How will we know what we're converting from?

On Oct 13, 2012, at 19:17, Chris Drackett notifications@github.com wrote:

why not convert to unicode in requests, just to make sure?

—
Reply to this email directly or view it on
GitHubhttps://github.com/kennethreitz/requests/issues/891#issuecomment-9406485.


IOW, either we require that you specify an encoding or we require Unicode.
Might as well do the latter. The broken part is the lack of docs.

On Oct 13, 2012, at 19:17, Chris Drackett notifications@github.com wrote:

why not convert to unicode in requests, just to make sure?

—
Reply to this email directly or view it on
GitHubhttps://github.com/kennethreitz/requests/issues/891#issuecomment-9406485.
correct url getting 400 status with requests

I'm not sure if this is a requests or requests-oauth issue, so I've opened it both places:

https://github.com/maraujop/requests-oauth/issues/40

basically I'm sending a request that gets a 400 status from requests. However, if I take response.url and request it again, I get the correct response. Odd?

``` python
response = self.client.request(method, url, data=request_params, allow_redirects=True)
response.raise_for_status()
>> 400

response = self.client.request(method, url, data=request_params, allow_redirects=True)
response = requests.get(response.url)
response.raise_for_status()
print response
>> correct response json
```


Why are you using requests-oauth, and not the built in oauth support?


because I was reading your docs ;)

http://docs.python-requests.org/en/latest/user/quickstart/#oauth-authentication


@chrisdrackett _cough_ check again ;)
python setup.py bdist_rpm fails

I fixed that by:
- Removing all but the 1'st line in LICENSE
- Deleting the two lines containing PYTHONDONTWRITEBYTECODE in setup.py

Python 2.7, Ubuntu 12.04 amd64


It requires there to be no license?

---

I've never even heard of this command before :)


If you're packaging this for a repository, I'm fairly certain that you're going to introduce a bug by removing `PYTHONDONTWRITEBYTECODE`. That was a specific bug fix for posting files. As for the LICENSE, why does that have to be truncated?


I had the same problem a while back.  I think the issue was that entire text of the license gets included, and the linebreaks cause each line of the license to be interpreted as config (somewhere within RPM packaging, .spec file maybe?).  I'm guessing it expects the license to be a single line of text.


LICENSE is included "as is" in the .spec file and this causes a parse error.

As for `PYTHONDONTWRITEBYTECODE` - I'm not an expert in RPM building but the bdist_rpm does byte compile everything and if it doesn't - it's an error. Maybe there's a flag to suppress that, but I can't find it. Maybe a post install script that does `find INSTALL_DIR -name *.py[co] -exec rm {} \;` will do the trick?


@sagarun might know about dist_rpm and the LICENSE issue.

Did we set PYTHONDONTWRITEBYTECODE in order to fix a test that was trying to upload `__file__`? I can submit a pull request that posts a normal text file instead, or works out a relative path to the original Python source file.


I just did a quick google search: http://www.rpm.org/max-rpm/s1-rpm-inside-tags.html explains that anything starting with a field-name has to be one line, e.g.,

```
Author: Kenneth Reitz
Copyright: ISC/BSD/etc.
```

But things like the description can be multi-line, e.g.,

```
%description:
 Foo
 Bar
 Bogus
```

Maybe there's a way to move the terms of the license into the `%description` tag then?


This is a bug in setuptools.
safe_mode not catching error on session.get

for this code

``` python
import requests
config = {'safe_mode': True}
# this url will get a connection reset
url = 'http://www.dapenti.com'
```

when set safe mode on requests.get, it worked:

``` python
r = requests.get(url, config=config)
print r.error
# ==> requests.exceptions.ConnectionError(socket.error(104, 'Connection reset by peer'))
```

but set it on session will raise a exception:

``` python
s = requests.session(config=config)
r = s.get(url) 
# raise ConnectionError: [Errno 104] Connection reset by peer
```


Good catch. It looks like it's due to issue #639. I fixed this but my patch hasn't been merged, yet.


As shown above we actually have two open pull requests to resolve this issue, so I'm going to direct any discussion that way. Thanks for the report @flytwokites!
Return server's SSL certificate in Response or Session

It's sometimes necessary to see not only boolean SSL certificate validation status but also SSL certificate contents. It would be nice to have a way to access the SSL cert as dict() available from Response or Session objects.

P.S. I can add that as I've written a module to do exactly that http://ipsec.pl/node/1068


`requests` is going through a pretty major refactoring right now. If you'd like to make a pull request with this change, I'm sure Kenneth will consider it. In the meantime, I'm going to close this issue. =)


What has the refactoring got to do with a feature request?


I'm sorry, that was a lousy explanation, I was pretty tired when I wrote it. =)

The long answer is that Kenneth is refactoring `requests` along [these lines](http://kennethreitz.com/the-future-of-python-http.html), which you can see in #895. This means that Kenneth is pretty unlikely to get to this any time soon (this issue is already a month old and I only just got to it, and I've got way more free time than he does). So Kenneth won't be adding this himself, at least, not for a while.

I also won't be making this change, partly because I suck at API design and partly because this looks to me like a 20 percent feature: the vast majority of people will never use it, and those that want it are plenty competent enough to make the change themselves. Obviously, Kenneth might think differently, but I doubt he'll look at this issue for a while.

For that reason, I suggested to @kravietz that he write the change himself and submit a Pull Request. The worst case scenario there is that Kenneth decides he doesn't want to change the API for this feature, and so @kravietz ends up maintaining a fork; best case, he likes it and merges it.

If you want a second opinion, please let me know and I'll nudge someone else to take a look at this issue. =)

**tl;dr:** No-one in the last month expressed any interest in making this change, Kenneth won't look at this for ages, and I have no plans to make this change, so I judged this issue as dead.


No need for a second opinion, I was just curious about the rationale. Thanks.


Has this been addressed in the last 3 years?


@jayfk I'm certain there's no other information about this other than what's in the issue.
Function like urllib.urlretrieve for downloading files

Hello,

I would like to request function for downloading files.
Something like:
`r = requests.get_file('https://example.com/big_file.zip', 'path_to_file')`

I'm looking for it mainly for downloading big files (over 1GB), without fully loading them to memory.

Best regards


You can write a file without loading it into memory currently:

```
r = requests.get('https://example.com/big_file.zip'', prefetch=False)

with open('file', 'w') as f:
    for chunk in r.iter_chunks():
            f.write(chunk)
```


Nice to know but still I would prefer short one-liner over 4 lines.
P.S. You probably should update you example from `requests.get_file` to `requests.get`


This isn't about lines of code — it's about concise, succinct code. There is little value and many disadvantages of having a library write to files directly on your behalf.


Thank you for reply. I'll leave this to more experienced than me and use your code.

`iter_chunks` not exist, I need to use `iter_lines` or `iter_content` with big `chunk_size`

Best regards


Sorry, I hadn't had any coffee yet :)


For latest version(Edit:  v2+) of requests:

```
r = requests.get('https://example.com/big_file.zip'', stream=True)

with open('file', 'wb') as f:
    for chunk in r.iter_content():
            f.write(chunk)
```


@amitt001 That should work in the current version v2.10.0, and indeed in all of the v2.X.X series release.
Support CJK parameters when post files

When I post files with CJK parameters I got this exception:

```
Traceback (most recent call last):
  File "/home/everbird/code/requests/tests/test_requests.py", line 358, in test_POSTBIN_GET_POST_FILES_WITH_CJK_PARAMS
    data={'some': '中文'})
  File "/home/everbird/code/requests/requests/api.py", line 98, in post
    return request('post', url, data=data, **kwargs)
  File "/home/everbird/code/requests/requests/safe_mode.py", line 39, in wrapped
    return function(method, url, **kwargs)
  File "/home/everbird/code/requests/requests/api.py", line 51, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/everbird/code/requests/requests/sessions.py", line 241, in request
    r.send(prefetch=prefetch)
  File "/home/everbird/code/requests/requests/models.py", line 529, in send
    (body, content_type) = self._encode_files(self.files)
  File "/home/everbird/code/requests/requests/models.py", line 365, in _encode_files
    new_fields.append((field, str(val)))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
```

For py2, `str` is actually `unicode` according to `requests/compat.py`. It is OK like this:

``` python
In [1]: str('a')
Out[1]: 'a'

In [2]: unicode('a')
Out[2]: u'a'
```

but it failed as below:

``` python
In [3]: str('中文')
Out[3]: '\xe4\xb8\xad\xe6\x96\x87'

In [4]: unicode('中文')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-4-a7400b671605> in <module>()
----> 1 unicode('中文')

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0: ordinal not in range(128)
```

In `requests/models.py` at line 363-365 the `str()` seems should be the builtin `str`, not `unicode`. So I wrote a test named `test_POSTBIN_GET_POST_FILES_WITH_CJK_PARAMS` and fixed it.

Hope it helps. 


What does CJK mean?


@piotr-dobrogost `CJK` is a collective term for `Chinese`, `Japanese`, and `Korean`. You can find more details [here](http://en.wikipedia.org/wiki/CJK_characters).


Thanks for the fix!
SSL validation doesn't consider subjectAltName

When performing SSL certificate validation, requests raises an SSLError when the domain name from the URL and the Subject CN don't match.  Many recent SSL client implementations (including major browsers) now respect the subjectAltName extension, and consider a certificate valid if that matches, despite a mismatching CN.  Support for this would be greatly appreciated.


`requests` doesn't handle SSL authentication on its own, [`urllib3`](https://github.com/shazow/urllib3) handles it on our behalf. Thanks for the suggestion!
Add function for clearing cookies from a session

Don't know if this is worth it, but it helps me on a project I'm working on.  It's a little nicer than handling the cookie object directly.


Hmm, I wonder if we can subclass CookieJar and add a `purge` method.


I forget if we're using the standard cookielib, but if so: http://docs.python.org/library/cookielib.html#cookielib.CookieJar.clear


Perfection.


Oh wow, that does it much better.  This is unneeded.
Thanks.
Only register callable items in lists

Prior to this, you could sneak a list of anything to register_hook and it
would accept it. This will check if the items in the list are callable before
registering them. Also added a regression test to make sure if this gets
changed it will be noticed.


Rebased to avoid any problems with merging.


Looks great! Waiting for travis...


Seems Travis likes it too.


Thanks!


No problem. Should have been written this way in the first place.
Keep-alive doesn't work over http proxy

Although, requests sends a `Proxy-Connection: keep-alive` header, it forcibly closes the connection after each requests.
I tracked down the issue's origin to models.py, how the connection is acquired:

``` python
if proxy and not any(map(_p.hostname.endswith, no_proxy)):
    conn = poolmanager.proxy_from_url(proxy)
```

vs

``` python
if self.config.get('keep_alive'):
    conn = self._poolmanager.connection_from_url(url)
else:
    conn = connectionpool.connection_from_url(url)
```

In the proxy case, a poolmanager is instantiated in every request, while without proxy the given `self._poolmanager` instance is reused.
I have a workaround for the issue via using `poolmanager.ProxyManager(self._poolmanager.connection_from_url(proxy))` call instead of `poolmanager.proxy_from_url(proxy)` if keep_alive is configured.

If you find my solution adequate, I prepare a patch for the issue in the upcoming days.


I just ran into this limitation myself, and came here to see whether I needed to open a ticket with this information!  I strongly support your proposed patch.


@Lukasa @shazow @kennethreitz opinions on this? It seems reasonable to me, but I'm not as familiar with urllib3 honestly.


+1 to the issue. The connection should definitely remain open across proxy requests. That's one of the most significant benefits of proxies--one connection for many requests across many hosts. :)


Any opinions on the proposed patch?


I can't think of a reason why you'd ever want new-connection-per-proxy-request, so I'd make keeping the same proxy connection the default behaviour.


So to flesh out @vlaci's proposed patch:

``` python
if proxy and not any(map(_p.hostname.endswith, no_proxy)):
    if self.config.get('keep_alive'):
        conn = poolmanager.ProxyManager(self._poolmanager.connection_from_url(proxy))
    else:
        conn = poolmanager.proxy_from_url(proxy)
```

should be feasible? I don't have a proxy to test against.

@vlaci if I'm misspeaking please pipe up. It'd even be awesome if you submitted the Pull Request to fix this.


My proposed solution works fine in our test system for a month. I'll make a pull request at the weekend then.


@vlaci awesome. Thanks for finding, reporting and fixing this.


Resolved by #982.


Issue reappeared in recent requests versions (tried with 1.1)


Related code in adapters.py:

``` python
    def get_connection(self, url, proxies=None):
        """Returns a connection for the given URL."""
        proxies = proxies or {}
        proxy = proxies.get(urlparse(url).scheme)

        if proxy:
            proxy = prepend_scheme_if_needed(proxy, urlparse(url).scheme)
            conn = proxy_from_url(proxy)
        else:
            conn = self.poolmanager.connection_from_url(url)

        return conn
```
Support for request body streaming

Sending a request with a large request body (i.e. a large file that is being multipart-encoded) will consume a large amount of memory.

It would be very helpful if it were possible to specify a file-like object (and the content-length separately) and request as well as its helper libraries read only the chunks that are needed to continue sending the request.

This may be related to https://github.com/shazow/urllib3/issues/51


The plan is to support generators. Work in progress :)


Thanks for letting me know. Just to confirm, did you close the ticket because you're not tracking features as issues or you're currently working on it? Is it appropriate to raise new features as issues in the future?


Oh, I should have mentioned #295 :)
Trouble posting some json

```
headers = {'content-type': 'application/json', 'accept' : 'application/json' }
my_config = {'verbose': sys.stderr}
return requests.post(url + '/doc/processText', data=PlainTextInputOptionsEncoder().encode(request), headers=headers, config=my_config)
```

The above is rewarded with a 500 from the tomcat hosting the service. The json looks OK when I dump it out with 'print'. Is there something I'm missing?
Python 3.3 compatible update

`httplib.cookiejar.DefaultCookiePolicy` changed its implementation of function `set_ok_verifiability` in Python 3.3

It now checks `request.unverifiable` instead of `request.is_unverifiable()`


Traceback without my modification:
http://pastie.org/4867827


@piotr-dobrogost I was thinking the same thing.


Well I think this modification has the least code change. Adding an `unverifiable` property may needs more code change. But anyway it's just a proposal to fix this problem.


From what I can see the code change using a property would be something like this (although I probably would include a docstring if doing this for real).  Looks about the same IMO.

``` Python
@property
def unverifiable(self):
    return self.is_unverifiable()
```


+1 for a property as opposed to changing `__getattr__`.


+1


property decorator is clearly a better solution.


@yegle care to update?


I've updated this pull request :-)
Installing Requests with PIP 1.2.1 and Python 3.1 encounters errors

Requests still installs, but setup.py seems to be running Python 2 code?

https://gist.github.com/3790404


Those are all just warnings. Requests vendorizes a package called chardet which is specifically for python2. It doesn't break python3, but it gives those horrible and ugly warnings that you saw. A package maintainer complained about the same thing (#832). #858 is also related but only in that @craigholm was hoping to find a way to conditionally import/install chardet.


The errors are benign, but they sure look like a problem. Unfortunately everyone who installs requests under Python 3.x will waste time researching this, until the find out it's not a problem.

It would be great if a solution could be found, but from the comments above it seems complex.


It is complex :)
Handle encoding of `None` when decoding unicode

If encoding is None, decoding will throw the following TypeError:

```
  File "/home/env/local/lib/python2.7/site-packages/requests-0.14.0-py2.7.egg/requests/models.py", line 825, in text
    content = str(self.content, encoding, errors='replace')
TypeError: unicode() argument 2 must be string, not None

```

If this is the case, attempt to run without any set encoding


Since you're handling this the same way as a `LookupError`, why not fold the two together, e.g.,

``` python
except (LookupError, TypeError):
    # A LookupError is raised if the encoding was not found which could
    # indicate a misspelling or similar mistake.
    #
    # A TypeError can be raised if encoding is None
    #
    # So we try blindly encoding.
    content = str(self.content, errors='replace')
```

Less code, less confusion.


@sigmavirus24 sounds reasonable!


:+1:


Thanks!