Raibek/requests-github-issues huggingface dataset:
Odd HTTPS issue

It appears that Requests isn't getting the correct SSL cert on my domain, for some reason.

```
➜  ~  python
Python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get("https://ultros.io")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: hostname 'ultros.io' doesn't match either of 'www.archivesmc.com', 'archivesmc.com'
>>>
```

The certificate itself is fine - if you visit [the site](https://ultros.io), then you do indeed get the correct cert.

![Certificate view: http://i.imgur.com/fVXe5YA.png](http://i.imgur.com/fVXe5YA.png)

This seems to be a fairly odd one. Server is nginx, any idea what's going on here?


My guess is that it's SNI related. Your webserver requires SNI-enabled clients (https://www.ssllabs.com/ssltest/analyze.html?d=ultros.io).


"This site works only in browsers with SNI support. " - is that what you mean?


Yes. Your webserver probably serves the "www.archivesmc.com" certificate for clients without SNI.


I'm unsure what you're saying here. The server itself supports SNI, is what I'm getting from that message, but Requests doesn't?


Exactly. That's what I'm guessing.


Ergh, that's problematic.

I'd assume it could be fixed, then, by setting ultros.io as my default nginx domain, but then Requests would fail to work on the archivesmc.com domain or the others I have on that server that use SSL.. May take some brain-crunching!

In the meantime, are there any plans to support SNI in Requests/Urllib3?


According to [this page](https://urllib3.readthedocs.org/en/latest/contrib.html), it's possible to use SNI with urllib3 - if I'm not mistaken, Requests is based on that library, right?


Confirmed fix.

``` python
>>> import urllib3.contrib.pyopenssl
>>> urllib3.contrib.pyopenssl.inject_into_urllib3()
>>> import requests
>>> r = requests.get("https://ultros.io")
>>> r
<Response [200]>
>>>
```


That fix is unnecessary. =)

If you simply install pyopenssl, pyasn1 and ndg-httpsclient, requests will automatically perform SNI for you. Alternatively, pip install requests with the 'security' optional dependencies: `pip install requests[security]`.

Thanks for your help @KoesterD!
Add custom line delimiter to iter_lines()

I saw a couple of requests that touched on this issue but, none that have hit it on the nose.  Has adding  a custom line delimiter functionality to `iter_lines()` been considered before? If so, what are the reasons for not including it?  If the feature has not yet been considered, I would not mind submitting a pull request for comment.

This feature would be useful in cases when, for example, newlines are a part of data and the stream is delimited with, say `<>`.


I don't believe it has. It's a pretty niche feature, but I'd be open to considering a PR with it.


Fixed #2295
Direct TLS pinning of .pem files possible? (No CA roots)

How can I pin a downloaded .pem TLS cert file for a specific server directly (self-signed certificate), without using CA roots?

Is this currently possible? If yes, can you please provide an example?

 I read http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verificationbut am not sure if this applies to what I'm trying to do:

<blockquote>You can also specify a local cert to use as client side certificate, as a single file (containing the private key and the certificate) or as a tuple of both file’s path:
<code>
requests.get('https://kennethreitz.com', cert=('/path/server.crt', '/path/key'))
Response [200]
</code>
</blockquote>



This is a question and as such belongs on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).


@sigmavirus24 I am asking about a pyhton-requests functionality. Can you please give me a simple yes or no answer if this is possible?


@HulaHoopWhonix We know what you're doing, we're just asking you to take the question to the appropriate location. GitHub issues are for feature requests, bug reports and code enhancements. They are not a forum for questions.


@Lukasa Ok I see what you mean. Any input for this question on stackoverflow would be appreciated. I posted it here:

https://stackoverflow.com/questions/26479039/python-requests-direct-pem-pinning-with-self-signed-cert
Fix #2279. Update layout css

Updated gratipay widget iframe in both sidebarintro.html and sidebarlogo.html . 


@syedsuhail Quick question, why'd you change the width?


I didn't notice the change already made in sidebarintro.html.  When I saw the gratipay widget was shorter in the requests page I tried adjusting the width. Then I understood the problem was in sidebarlogo.html. I checked the html build and didn't see any problem with the current width. So I kept it as such.


Fair enough, suits me. :cake: Thanks!


Thank you very much. :smiley:
Fix #2288 . Change urllib3 and chardet workflow

 This modified workflow first tries to clone the urllib3 or chardet and then removes it from requests/packages if successful. If the cloning is unsuccessful then the packages are not removed.


Thanks for this @syedsuhail! :cake:


Happy to contribute. This is my first successful merge. :smiley: 


@syedsuhail I'm glad you chose to contribute to requests!
Revisit concerns in #2244

See https://github.com/kennethreitz/requests/pull/2244 for more context:

Specific concerns are:
- Should redirects handled by the cache count against the max number of redirects? (e.g., if we set a max of 30 and find 2 redirects in the cache, the max number of redirects a user could encounter after that would be 28)
- Should history be some-how generated by the redirect cache? (i.e., should we be making responses and adding them to the final  response's history?)
- Should a separate error be raised for the case where there's a loop in the cache? (When documented this would help the user realize they're stuck in a cached loop and that they may want to clear it to retry using the live site. Note: this also affects the API of `Session#resolve_redirects`.)


@Lukasa is it still worth keeping this open?


I still hate the redirect cache, but I think we should talk about it at PyCon. In particular, we should talk about whether we should yank it out and instruct people to use CacheControl instead.
res.json ist not a function under debian

Hi

If I do a HTTP request resulting in a JSON, I can access it directly in Debian 7.6

```
    res = requests.get('url')
    res.json['member']
```

In fact, when I call it as a function as described in the docu, it throws an error:

```
     res.json()
     TypeError: 'dict' object is not callable
```

So, on my system res.json is always a parsed JSON (a dict), while on other systems this is not the case.

My Python an rquests version: 
Python 3.2.3 (default, Feb 20 2013, 14:44:27) 
[GCC 4.7.2] on linux2

requests==0.12.1


Thanks for reporting this!

This is not a requests bug. The problem lies with your machine. The current requests version is 2.4.3, as can be seen [on PyPI](https://pypi.python.org/pypi/requests). The version you have installed, 0.12.1, was released more than two years ago. Understandably, the documentation for 2.4.3 does not apply to two year old versions.

You should update requests. If that doesn't work then Debian are packaging an extremely old version, and you should complain to them, and then install requests from pip.


thanks, works fine with the current version

```
      pip install --upgrade requests
```
Update sidebarintro.html

Fix issue #2279. "Gittip" changed to "Gratipay". Tip button extended from 48px to 60px.


@kennethreitz you okay with this?


<3


Yay, my first merge. :) 


Thanks @morganthrapp! :cake: 


Shouldn't the URLs be updated to https://gratipay.com/kennethreitz/ ?

Redirect from old domain works for now for the link, not sure about the iframe.


It probably should.
On Oct 15, 2014 7:47 AM, "Arjen" notifications@github.com wrote:

> Shouldn't the URL be updated to https://gratipay.com/kennethreitz/
> (redirect from old domain works for now) ?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2282#issuecomment-59193588
> .


Gittip redirects to Gratipay so it isn't urgent
Transfer-Encoding: chunked; API to get trailers?

Is there a way to process trailers using Requests?

http://tools.ietf.org/html/rfc7230#section-4.1.2

Adding a trailer, (e.g. Content-MD5) is
especially useful to detect incomplete
streamed reponses from an apache server,
otherwise its difficult (impossible?) to write
a robust client for large server responses.


I'm afraid `httplib`, upon which requests is built, provides no API for obtaining trailers. This means we can't provide one either.

We are thinking of rewriting `httplib`, which would enable us to provide such support. In the meantime, I recommend you request the feature from the CPython developers.

Sorry we can't be of more help!


Is there an existing thread somewhere on the status of
chunked trailers support for httplib in CPython and where
to post the request for it?


I did a quick search but found nothing. Trailers are a relatively uncommon feature, and I could find no request for the feature absolutely anywhere.

I recommend opening it on bugs.python.org, or possibly sending a mail to python-list.
Gittip renamed to Gratipay, layout needs updating

Gittip [was renamed to Gratipay](https://github.com/gratipay/inside.gratipay.com/issues/73), and now the sidebar button is cut off:

<kbd>
![donation button cut off](https://cloud.githubusercontent.com/assets/46775/4617065/274b276a-52fa-11e4-9e0c-16544d473696.png)
</kbd>

Looks like some CSS needs adjusting.

For extra bonus points: adjust the link text in the sidebar too.
apparent_encoding() hangs with high CPU utilization

While processing a large set of requests, `apparent_encoding()` works well for quite a while but eventually hangs. All I have at this point is a traceback:

```
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/models.py", line 637, in apparent_encoding
return chardet.detect(self.content)['encoding']
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/__init__.py", line 30, in detect
u.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/universaldetector.py", line 128, in feed
if prober.feed(aBuf) == constants.eFoundIt:
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/charsetgroupprober.py", line 64, in feed
st = prober.feed(aBuf)
File "/opt/wsgi/.../ENV/lib/python2.7/site-packages/requests/packages/chardet/sjisprober.py", line 74, in feed
self._mContextAnalyzer.feed(aBuf[i + 1 - charLen:i + 3
```

I appreciate anything you can do to help.


This is a duplicate of https://github.com/chardet/chardet/issues/29 which no one (who is remotely interested in the bug) has had the time to investigate or fix yet. This isn't a bug in requests.


This is also creating problems for me.
Any work around?


@dav009 two options:
1. Use cChardet instead of `apparent_encoding`
2. Set a custom encoding if you know it to prevent requests from needing to use `apparent_encoding`


@dav009 one other alternative, help chardet fix the bug on their side.
make .raw more file like

This is kinda half baked, but maybe there is a pattern that fits what I am looking for: 

> > > r=requests.get(url)
> > > r.raw.name  # AttributeError: 'HTTPResponse' object has no attribute 'name'

just like:

> > > fd = open('foo.txt')
> > > fd.name
> > > 'foo.txt'


@CarlFK what would that even return?


Beyond that, this belongs on urllib3. The `raw` attribute is a urllib3 HTTPResponse. I also see no reason to provide this. It may have a few file-like methods, but there's no reason for it to be more file-like.


The easy answer is url.
maybe parse it to look for something file namy like.

or look in the headers for something - like my server side code:
response['Content-Disposition'] = 'inline; filename=playlist.m3u'

> > > r = requests.get(url)
> > > r.headers
> > > CaseInsensitiveDict({'content-disposition': 'inline; filename=playlist.m3u',

But I see your point about urllib3, so yeah, never mind.


You realize you could easily write your own wrapper class for this right?

``` python
class FileLikeHTTPResponse(object):
    def __init__(self, raw):
        self.raw = raw
        self.name = self.find_name()
        # etc.
    def read(self, amount=None):
        return self.raw.read(amount)
    # etc
```
Attempt at #1956 DO NOT MERGE

My attempt at fixing this issue, not sure its great


@mikecool1000 why did you close this?


I just didn't want try to submit work I am unsure of

Sent from my iPhone

> On Oct 12, 2014, at 7:16 PM, Ian Cordasco notifications@github.com wrote:
> 
> @mikecool1000 why did you close this?
> 
> —
> Reply to this email directly or view it on GitHub.
update python-requests.org to SSL

it's now free if you use Cloudflare for DNS


Technically we have SSL, but it's only for ReadTheDocs.org so there's a Certificate Name mismatch. We could proxy a certificate from Heroku https://docs.readthedocs.org/en/latest/alternate_domains.html#cname-ssl 

```
21:17.03  ericholscher he can proxy back to RTD from heroku as well
21:17.06  ericholscher if ya'll want
21:17.12  ericholscher for the ssl term
```


I don't think this really matters. 


@kennethreitz: it matters because of https://github.com/requests/requests/issues/4885
form url encoding data

Hi I have the following data structure which needs to be sent to a web-service as form-urlencoded:

```
data = {'Properties': [{'key': 'KeyLength', 'value': '512'}], 'Category': 'keysets', 'Offset': '0', 'Limit': '100'}
I'm posting it like this:
```

```
req = requests.post('http://server1/ws1/api/data/filters/', data=data)
```

This is how it should look received after its been encoded and decoded:

```
Properties[0][key]=KeyLength&Properties[0][value]=768&Category=keysets&Offset=0&Limit=100
```

However it seems end up like this:
``
Category=keysets&Limit=100&Properties=[{'value': '512', 'key': 'KeyLength'}]&Offset=0

```
Here's how it looks in the chrome developer tools which works fine from the browser.

![image](https://cloud.githubusercontent.com/assets/555399/4567621/058b3176-4f33-11e4-8eb2-30c0dbbd7dc7.png)


Can somebody please advise what I'm doing wrong.
```


You already asked this question on [StackOverflow](https://stackoverflow.com/questions/26266664/requests-form-urlencoded-data) which is the appropriate forum. This is a common problem and since I don't have the time to answer your question there now (and didn't look since you posted it 40 minutes ago), I'm sure someone else will link you to the relevant answer. Please be more patient with StackOverflow. Our community there is vibrant and will usually answer your question in less than a day.


OK - Thanks. I re-wrote the question on StackOverflow to be more descriptive and because I've sort of worked out what going wrong. However, still can't get it to work. I also tried 'werkzeug.MultiDict' but couldn't get that to work either.
Move utility functions from requests.utils to requests-toolbelt

# Reasoning

requests is an _HTTP_ library, not an HTML library. Some of these utility functions are far more focused on HTML and a Response's body than on HTTP. With that said, they don't belong in requests proper. Instead they belong in a third-party library and currently the best place for that is requests-toolbelt.
# Path to removal

Step 1:
- [x] Add `DeprecationWarning`s to all of these functions in the very next possible release of requests (either 2.4.3 or 2.5.0) (See PR https://github.com/kennethreitz/requests/pull/2309)

Step 2:

Prior to releasing 3.0, remove the following functions and migrate them to [requests-toolbelt](/sigmavirus24/requests-toolbelt). Allow requests-toolbelt to release a version first and then release 3.0.
- [x] `get_encodings_from_content`
- [x] `get_unicode_from_response`
- [ ] more functions?


+1000000


Why `get_unicode_from_response`?


We already provide the unicode in the response as `r.text`.


Back in the day, we had a function that would convert any Request object into a curl command string.

...lol


Yeah, people have recreated that time and again. That's not ending up in the toolbelt frankly. On the other hand, if someone wrote a function to convert it to an HTTPie command string... I might consider that ;)


I think we can close this.
ImportError: cannot import name urlencode

I am running Python 2.7.8. I get the following error from `import requests`:

```
Traceback (most recent call last):
  File "request_get.py", line 1, in <module>
    import requests
  File "/usr/local/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/local/lib/python2.7/site-packages/requests/utils.py", line 25, in <module>
    from .compat import parse_http_list as _parse_list_header
  File "/usr/local/lib/python2.7/site-packages/requests/compat.py", line 7, in <module>
    from .packages import chardet
  File "/usr/local/lib/python2.7/site-packages/requests/packages/__init__.py", line 3, in <module>
    from . import urllib3
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/__init__.py", line 10, in <module>
    from .connectionpool import (
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 37, in <module>
    from .request import RequestMethods
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/request.py", line 4, in <module>
    from urllib import urlencode
ImportError: cannot import name urlencode
```


``` pycon
$ python
Python 2.7.8 (default, Aug 24 2014, 21:26:19)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from urllib import urlencode
>>>
```

Can you give us more information?
- How did you install requests?
- What system are you running on?
- What happens when you do `python -c 'from urllib import urlencode; print("Imported")'`?
- Do you have a file in the same directory as `request_get.py` called `urllib.py`?


@sigmavirus24 Thanks for giving me pointers for troubleshooting. The file I was originally using was haphazardly called 'urllib.py'. Once I ran Requests from a different directory it worked fine.


@evanemolo it was a lucky guess. :)


The error message for this is really bad. I wish they'd make it more clear
that you're shadowing a module name.

On Saturday, October 4, 2014, Ian Cordasco notifications@github.com wrote:

> @evanemolo https://github.com/evanemolo it was a lucky guess. :)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2264#issuecomment-57923709
> .

## 

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com


@kevinburke it's doubtful that this will be updated in 2.7 and in 3.x it's a non-issue. I tried adding `from __future__ import absolute_import` and that doesn't fix it either. There's not much we can do here.
Fixed #2205

Hello.
I made some fixes to `get_unicode_from_response` fixing #2205.
It now tries every encodings from meta tags by calling `get_encodings_from_content`.
Because encoding charset is case-insensitive by RFC 2616, encoding charset is appended to `tried_encodings` in lower case.


This looks great @daftshady, thanks! :cake: I'll let @sigmavirus24 review, but I'm :+1:.


I have one concern. Once that's addressed I'll be :+1:.


@sigmavirus24 I agree with you. It's fairly reasonable to use a set here. I added another commit.


Looks great now! Thanks @daftshady!


This functionality was actually intentionally removed from Requests, and the documentation was accidentally not updated appropriately. 

Let's fully remove that function in v3.0.0.


#2266
Ignore proxy or always use 80 port

Hello!
I use lib and get requests with proxy on port 8080.
Example:
r=requests.get('http://www.rambler.ru/', proxies={'http:':'http://login:pass@x.x.x.x:8080', 'https:':'https://login:pass@x.x.x.x:8080'}, timeout=5)
Outgoing connections to port 80 on the server closed. And this request I freezes on this server. But running on the server where the 80 ports are open.
Also I see the blocked connection firewall on port 80.


The proxies parameter should be:

``` python
proxies = { 'http': 'http://login:pass@x.x.x.x:8080', 'https': 'https://login:pass@x.x.x.x:8080' }
```

You need to move the colons out of the first and third fields.


@Hasimir is correct. Please consult the documentation before raising bugs. =)


Sorry)
HTTP Authentication Any

Should `requests` supports the "ANY" authentication method for http authentication?

PHP curl supports this method: http://php.net/manual/en/function.curl-setopt.php (please search for CURLOPT_HTTPAUTH).

This method will support http authentication for both Basic and Digest. It will send the header with Basic first, and try for Digest if the server requests for Digest.

Because requests is for human, normally the client should not care about Basic or Digest.

What do you think?


It's not clear to me that it's worthwhile adding this to requests proper. However, the [toolbelt project has exactly what you're looking for](http://toolbelt.readthedocs.org/en/latest/user.html#guessauth).


Thank you, this is exactly what I am looking for.

P/S: In toolbelt's documentation, I think there is a typo:

> This requires an additional request in case of **basic auth**, as usually basic auth is sent preemptively.

This should be _Digest auth_.

Thanks,


Nope, that's not a typo, it's just a little unclear. It means _when compared to requests default basic auth_, it requires an additional request. This is because normally requests doesn't wait for a 401 for basic auth, it just emits the header straight away. GuessAuth doesn't do that.


Oh, that way will make 2 requests for both of Basic and Digest auth?

If it's true, I think I should create an issue for GuessAuth.

Do you know is there a special reason that GuessAuth has to do that?


@tarzanjw The short answer is because it doesn't know which header to send until it gets the 401 challenge. It's not _necessary_ that it do this, it could send Basic by default.


It _could_ but it won't (wearing my hat as toolbelt maintainer). Also, this isn't the place to discuss this.


Thanks for you reply,

If I use GuessAuth, it will sent the WWW-Authorization header at 2nd request (the one after 401 response), right?

If the server support Basic authentication, and the GuessAuth send the WWW-Authorization header at the first request, it will take us just 1 request only.

Do you think that 1 is better than 2 ?


It _might_ save us one round trip, but it might also cause the server to return a 403 Forbidden. That would be bad. 


It's okay, thanks for you reply.
Add json parameter

Closes #2025


@Lukasa @kevinburke care to take another look at this?


LGTM


omg I'm so excited 

:heart: :sparkles: :cake: :rose:  :cake: :sparkles: :heart:


@willingc thank you so much for your hard work on this. :cake: (Those emoji from @kennethreitz are for you ;))


Thank you @sigmavirus24 for your support and mentoring :guitar: :sunrise: :evergreen_tree: , @lukasa for the review :cat: :cat2: :smile_cat:  , and @kennethreitz for the community atmosphere :camera: :v: :guitar:


@willingc thank YOU, and you're welcome :)
Add json upload

Closes https://github.com/kennethreitz/requests/issues/2025

TODO:
- [ ] check `TODO`s
- [ ] more tests?
- [ ] `application/json` and files upload


@gagoman someone had been working on this for a while and stalled. I've opened a PR on their behalf. Did you read the entirety of the issue?


Yes, I had read an issue and there were no progress since June, so I've decided to implement it.


@sigmavirus24 reopen if it is still required.


@gagoman the discussion said the work would be slow but that two people were owning the work. The proper etiquette would have been to ask if they were still working in it. Thanks for your contribution, but we'll be continuing forward with their work.


@sigmavirus24 got it.
SSLError (hostname mismatch) on forge.ocamlcore.org

Using the latest version (2.4.1):

```
>>> requests.get('https://forge.ocamlcore.org/', stream=True)
```

Throws:

```
SSLError: hostname 'forge.ocamlcore.org' doesn't match either of 'lists.ocamlcore.org', 'ocamlcore.org'
```

The same URL shows no SSL issues in firefox or chrome, and curl also thinks it's fine:

```
$ curl -Iv 'https://forge.ocamlcore.org/'
( ... )
* Connected to forge.ocamlcore.org (87.98.154.45) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_DHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate:
*   subject: E=<redacted>@gmail.com,CN=forge.ocamlcore.org,C=CH,OID.2.5.4.13=ngEw1eS9i0H4lis9
*   start date: Oct 01 20:21:04 2013 GMT
*   expire date: Oct 02 00:48:22 2014 GMT
*   common name: forge.ocamlcore.org
*   issuer: CN=StartCom Class 1 Primary Intermediate Server CA,OU=Secure Digital Certificate Signing,O=StartCom Ltd.,C=IL
> HEAD / HTTP/1.1
> User-Agent: curl/7.32.0
> Host: forge.ocamlcore.org
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
( ... )
```

I'm afraid I don't know enough about SSL certs to debug this any further, but I'm assuming that requests is missing something if firefox, chrome & curl all accept the cert.


This is almost certainly an SNI issue. I don't have access to a shell right now to confirm it, but I suspect you're using Python 2.

In that case you need to install three new packages. You can either do that by reinstalling requests with the `security` extension (`pip install requests[security]`) or by installing them explicitly: `pip install PyOpenSSL pyasn1 ndg-httpsclient`.


Thanks for the tips - I had no idea about the security extension. I'm afraid it took me too long getting the native extensions installed, and in that time the certificate has expired anyway! But I'll take your word for this, and reopen if I need to.
A fix for #1979: repeat HTTP digest authentication after redirect.

This solves the following scenario:
(1) 401 (Digest authentication begins)
(2) 302 (Authenticated, response redirects to another endpoint)
(3) 401 (New endpoint also requires authentication)


Hey @yossigo thanks for the pull request!

While you're here, could you fix something that I think is a bug? On [this line](https://github.com/yossigo/requests/commit/c28da22e9c42e22b303bb07da434ce65e10c0cb2#diff-9f3a95293a5d26032b1c588167760362R193) could you change it to

``` python
setattr(self, 'num_401_calls', num_401_calls + 1)
```

We still want to check to prevent users from being caught in an endless challenge state with a server.

Right now, I'm trying to decide if I think `handle_302` should check the response code before resetting the attribute because it will be called regardless of whether a 302 was received or not.

Also, @yossigo what do you think about setting `num_401_calls` to 1 in `handle_302` instead of deleting the attribute? It would be much simpler. Also, there's no need to return `r` from `handle_302`.


It seems like the new 302 handler will actually zero out the count every request. That might be deliberate, but we should probably make sure it only gets called on redirects.


I've cleaned up a bit the original commit (which was too hasty).  Now avoiding possible looping around 401s and will handle all redirect responses.


I think this is ready to merge. Sorry I didn't see your update @yossigo 

Thanks for this!


:sparkles: :cake: :sparkles:
to use ssl connections ssl varialble must be differ than None.

Hello, I've get the next problem

   response = requests.post(self.url("translate"), data=data)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/api.py", line 92, in post
    return request('post', url, data=data, *_kwargs)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/api.py", line 48, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/sessions.py", line 451, in request
    resp = self.send(prep, *_send_kwargs)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/sessions.py", line 562, in send
    r = adapter.send(request, *_kwargs)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/packages/urllib3/connectionpool.py", line 722, in _validate_conn
    conn.connect()
  File "/home/andrew/.config/sublime-text-3/Packages/RusVariablesTranslator/requests/requests/packages/urllib3/connection.py", line 169, in connect
    self.sock = ssl.wrap_socket(conn, self.key_file, self.cert_file)
AttributeError: 'NoneType' object has no attribute 'wrap_socket'
....


Hey @kostyll can you tell us what version of Python you're using?


py3.3.3 - interpreter in sublime text 3. but I don't know how to import ssl(_ssl) module .... in sublime text 3 under linux python interpreter is compliled without this module ...


This is not really a bug, it's just what happens when you don't have an SSL module in your Python distribution. It looks like ST3 compiles without it, which isn't particularly surprising.


Thanks. I've already read about it. But how can I get this module (_ssl) supported in my ST3 ? I've read http://sublimetext.userecho.com/topic/50801-bundle-python-ssl-module/#comment_165820 but I've not understood the mechanism of including this module manualy yet.


This is not a bug in requests or urllib3. If you need help with this @kostyll you should see if there are any ST3 users who can help you with this. Better yet, I would advise not using the vendored copy of Python with ST3.


Thank you guys!) But I went another way - I've fount repo https://bitbucket.org/klorenz/sublimessl which contains implementation of binding of linking ssl library. The example of usage is you can find in my repo : https://github.com/kostyll/my-sublime-text-3-ya-translate-plugin which contains simple ST3 plugin. Good luck !)
fix #2247

We have to pass urllib3 the url without the authentication information,
else it will be parsed by httplib as a netloc and included in the request line
and Host header


This _looks_ right, but it's hard for me to know because URLs are a crazy mess. I'll let @sigmavirus24 review this, he knows URLs better than I do.


So if we used a better (read: less forgiving) URL parser library we wouldn't need to split on `@` (like we do [here](https://github.com/kennethreitz/requests/pull/2249/files#diff-5956087d5835a57d9ef6fff974f6fd9bR687)). In principle this works fine. In reality, we should get something like `rfc3986` into acceptable shape for @shazow or just make the `Url` object in `urllib3` more RFC compliant (whichever works better) and use that for parsing the URL and reconstructing it. In short, you have 5 major components of the URL:

```
{scheme}://{authority}{/path}{?query}{#fragment}
```

And `authority` which we're dealing with right now has 3 sub-components so a URL would look like:

```
{scheme}://{userinfo@}{hostname}{:port}{/path}{?query}{#fragment}
```

`rfc3986` would split this up and allow you to replace `userinfo` with `None` and then reconstruct the URL. For a quick fix, this is great. I'd rather not have so much URL/URI parsing logic in requests though, this is an HTTP library not an HTTP + URL + ... library.


+∞. I think it would be best to have URL objects that are immutable and an API like URL.replace(userinfo=None, fragment=None)


@t-8ch that's roughly `rfc3986`'s API, except that I don't think I call it `replace` but maybe I do.... I'll double check later.


@sigmavirus24 Please open the specific issue with urllib3's url parser, no need to be backhanded. :)


Sorry @shazow, it wasn't meant to be back handed. I'll pull together the list of things the object is missing and make an issue with it tonight.


Thanks. :)


What's the status of this? 


@kennethreitz :+1:
speed limit and time out

requests doesn't have speed limit? 
curl like this: --limit-rate
wget like this: --limit-rate


Correct.

Requests simply doesn't have this low-level of control over the HTTP stack except in one specific situation: when sending a chunked upload, which we handle ourselves. In all other cases `httplib` is responsible for sending our data, and _it_ is the one that lacks a speed limit.

Without a patch to `httplib` we cannot add such a feature. Even with a patch, that is a sufficiently low-level feature that we probably wouldn't add it anyway.

I'm sorry we can't be more helpful. =(
Source code clarification sought

I'd like some clarification as to whether the following code in [requests/api.py](https://github.com/kennethreitz/requests/blob/master/requests/api.py#L47-L48) is correct.

```
def request(method, url, **kwargs):
    """Constructs and sends a :class:`Request <Request>`.
    Returns :class:`Response <Response>` object.

    :param method: method for the new :class:`Request` object.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
    :param timeout: (optional) How long to wait for the server to send data
        before giving up, as a float, or a (`connect timeout, read timeout
        <user/advanced.html#timeouts>`_) tuple.
    :type timeout: float or tuple
    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
    :type allow_redirects: bool
    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.

    Usage::

      >>> import requests
      >>> req = requests.request('GET', 'http://httpbin.org/get')
      <Response [200]>
    """

    session = sessions.Session()
    return session.request(method=method, url=url, **kwargs)
```

It creates a temporary session but does not call `session.close()` when the session goes out of scope. Is this correct and intended? Will this leak resources as the adapters associated with the session are not closed? Could you clarify if this is correct?


This is correct.

`session.close()` is not mandatory. The only resource that needs to be worried about are sockets, and sockets that are leaked are closed by the Python garbage collector. There is no need to explicitly clear a `session`.
HTTPS Session Resume!!

This question might not be on the right project. It seems to me that the request library during HTTPS communication does not do SSL Resume. It would be nice to have SSL resume feature for optimization. May be I should ask this question to urllib3 forum?


I'd love this feature, it'd be awesome. Unfortunately, we're not in a place to do it at the moment.

The standard library has an [open feature request](http://bugs.python.org/issue8106) to add support for session resumption. No-one has yet done the work here, so the standard library won't support it. PyOpenSSL also doesn't have support, so we can't do it outside the standard library either.

If support ends up in either of those projects we can absolutely optionally use it, and I'd love to have it, but we need to get it supported there first I'm afraid.


The new release version (0.14) of pyOpenSSL has opened number of callbacks from the subset of OpenSSL library. The particular API called 'set_session' can be use in pyOpenSSL.py (/requests/packages/urllib3/contrib/pyopenssl.py) to enable SSL session reuse. There are two tasks we need to do to make it happen  I think
1. Store the initiated established session for particular host:port so that we can reuse the same session for the subsequent request.
2. Use global SSL context or Context associated for particular host:port. I don't know which way to go

Let me know if there are any plans in near future to add SSL resume support.


If you're interested in this feature, I recommend opening a feature request on urllib3, which is where the work needs to be done. =)


Thanks, will do.


@xnulinu would you mind linking the feature request here?


The feature request link I have opened on urllib3 library is : https://github.com/shazow/urllib3/issues/590
raise RuntimeError when a single streamed request calls *iter methods th...

In response to issue #2240, add a check to see if the content was "consumed"


Let's have a subclass of `RequestException`. My proposed name is `BodyConsumedError`: @sigmavirus24, do you have a better one?


EOFError?


I think `BodyConsumedError` is preferable to `EOFError` and `StreamConsumedError` might be even more obvious in this case.


Thoughts on making whichever error name we choose multiply inherit from `RequestException` and `TypeError`? People who are doing this may already be catching `TypeError`, if we magically change this on them, it's a breaking change


Yeah, I can accept that. And `StreamConsumedError` is the best of the bunch so far.


@tijko so could you update this PR to add the described exception to `requests/exceptions.py` and then use it here where you're raising the `RuntimeError`? Thanks in advance


@sigmavirus24 looks like @tijko updated this as requested


@kennethreitz looks good to me
Multiple calls to iter* fail with unhelpful error.

If you call the `iter*` methods more than once for a single streamed request, it fails with an extremely unhelpful error:

``` python
>>> r = requests.get('http://www.google.com/', stream=True)
>>> [x for x in r.iter_content(1024)]
>>> [x for x in r.iter_content(1024)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/requests/utils.py", line 341, in iter_slices
    while pos < len(string):
TypeError: object of type 'bool' has no len()
```

I think it's fine that you can't call the iter\* methods twice on a streamed response, but we can easily catch this situation so we should.


I'm making a pull request with a `RuntimeError` being raised.  What kind of exception would you ideally like to throw here?  


I'm going to close this to centralise on #2241.
ssl and multiple subdomains


When I attempt to retrieve a url like, https://ia700306.us.archive.org/11/items/MITclassical_mech/

```
requests.get("https://ia700306.us.archive.org/11/items/MITclassical_mech/")
```

I am getting an exception

```
lib/python2.7/site-packages/requests/adapters.pyc in send(self, request, stream, timeout, verify, cert, proxies)
    418         except (_SSLError, _HTTPError) as e:
    419             if isinstance(e, _SSLError):
--> 420                 raise SSLError(e, request=request)
    421             elif isinstance(e, ReadTimeoutError):
    422                 raise ReadTimeout(e, request=request)

SSLError: hostname 'ia700306.us.archive.org' doesn't match either of '*.archive.org', 'archive.org'
```

But Chrome shows the domain as authenticated with SSL. Is this a bug or by design? 


It's neither. =)

We are correctly validating the certificate that was passed to us, but `archive.org` passed the wrong certificate. This is because it uses an extension to TLS called SNI (Server Name Indication) to work out which TLS certificate to present. Python 2.7's standard library doesn't have support for this extension, so out of the box requests can't use it.

If you're on the most recent version of requests (2.4.1) you can re-install requests with `pip install requests[security]` to get these features. If not, you need to install the following extra libraries:
- `pyOpenSSL`
- `ndg-httpsclient`
- `pyasn1`


Excellent. That worked

Could this SNI issue be detected and have the above instructions added as part of the exception?


The best we could do is catch the verification error and say, on Python 2, "this may be a result of lacking SNI". We can't know for sure because it looks like a standard certificate validation error.


Could a friendly bootup message alert the user that they are running on Python 2 without SNI enabled and that many sites will have certificate errors unless they install the following packages ... ?

IPython did something similar for platforms that had a misconfigured readline/libedit.


## Unlikely. Urllib3 added on-by-default warnings that significantly angered our users. We won't be reproducing that mistake at all

Sent from my Android device with K-9 Mail. Please excuse my brevity.


@sigmavirus24 is right. We are still arguing about whether or not that was ok.
Support bytestring URLs on Python 3.x

Hi there folks.

Currently `prepare_url` will call `unicode` or `str` on the url arg depending on the python version. This works fine for most cases, but the one case it trips up on is bytestrings on python 3.x as the string representation of these is `"b'http://httpbin.org'"`. Eventually this will surface as an `InvalidSchema` exception.

I find this to be completely unexpected, and I'd imagine it's not something that's been done intentionally.

Technically this a breaking change. The possibility of passing non-strings to `prepare_url` is undocumented and untested, but regardless it may be better to go about fixing this in a different way, that's your call.


Thanks for this!

I think we like the ability to pass non-strings to `prepare_url`. It allows people to use custom url-building classes without running into trouble. It's worth noting that you change also breaks Python 2 behaviour: previously a Python 2 `str` would get lifted to a Python 2 `unicode`, which it now doesn't do.

I think we can get around this by simply special-casing string types. The logic is really:

```
if type is unicode, leave unchanged
else, if type is bytes, decode bytestring to unicode
else, if type is anything else, call the 'to unicode' method
```

I think that's really the logic we want here. @sigmavirus24, thoughts?


I agree with @Lukasa that this is the behaviour we want. We absolutely want `unicode` urls because on Python 3, we can handle IRIs. We would need to adopt something that implements RFC 3987 to support it on Python 2, but once we did, we would be able to support them there too. For example, http://☃.net should be supported. (Your browser, and Python 3 should properly "encode" that to http://xn--n3h.net/.)


Ah good point, forgot about that use case.
Have updated the branch, I believe it's doing what it should be now, how's that?


:heart: @buttscicles 

@Lukasa this looks okay to me. Thoughts?


:cake: Make it so.


Let's not document this :)
Would it make sense to merge responses into requests ?

Hi,

requests is great, but testing code using it is a bit difficult without scaffolding. The [responses project](https://github.com/dropbox/responses) is fairly neat, but since it needs to access internals of requests, wouldn't it make sense to include the feature in requests ?


## No. There are plenty of libraries that access our internals. Most (if not all) do not belong in requests core.

Sent from my Android device with K-9 Mail. Please excuse my brevity.


As @sigmavirus24 says, lots and lots of projects hook into requests and we very definitely don't want to include them in requests. =)


If you want to discuss this further, email me
when setting the Accept-Encoding header to None, it is still sent as 'identity'

I am passing these headers to request:

headers = {'Content-Type': 'application/something+xml',
           'Accept': 'application/something+xml',
           'Host': host,
           'Expect': '100-continue',
           'Connection': None,
           'Accept-Encoding': None}

and then using:

r = requests.put(my_url, data=strata_xml, headers=headers, auth=(user, pwd), verify=False)

the REST service on the other end does not like this header at all.  Using nc, I see it sending:

PUT /myurl HTTP/1.1
Accept-Encoding: identity
Content-Length: 1200
Accept: application/something+xml
User-Agent: python-requests/2.4.1 CPython/2.7.8 Darwin/13.3.0
Host: localhost
Expect: 100-continue
Content-Type: application/something+xml

I believe that https://hg.python.org/cpython/file/d047928ae3f6/Lib/http/client.py#l1086 may offer some clue there as to why this header is not being deleted when I set it to None, but I don't know how to work-around that.

Thanks.


Essentially http.client adds "Accept-Encoding: identity" if there is no "Accept-Encoding" header present in the prepared request and this skip_accept_encoding option is not set (to clarify further)


@vdanen we remove `None` before passing the headers on to `urllib3`. `urllib3` does not do the same processing that we do on headers before passing them on to `httplib`/`http.client` (as it shouldn't). There's no way to pass this on to urllib3 in such a way that it can also tell `httplib` that. Is there any `Accept-Encoding` value that this server won't balk at?


Not that I have found, no.  It doesn't like that header at all it seems.  I've also tried to use a generator to force it to be chunked (which is the only way this header doesn't get sent it seems), but the server really didn't like that.

I've asked if there is a way they can fix their end to accept/discard/ignore that header without dying, but I probably won't hear back on that for a bit, and I may not be the only one to run into this problem which is I reported it.


That's really fairly terrible. `Accept-Encoding: identity` is _always_ valid, the RFCs say so. It should be utterly harmless to send it along.

Otherwise, removing this requires us to replace httplib. That's a substantial bit of work. =(


Yes, I see that.  And I certainly wouldn't expect to have to gut this just to make it work for my use-case which seems to not be that standard.  Thanks for the info!  If it's not possible to work-around this in any reasonable way, then there probably isn't a point in keeping this open.


😢 

Need the same thing for Connection:keep-alive - don't send it. :-(

This is an issue from 2014 that is not related to the current comments. To avoid reviving a long dead issue, I'm locking the conversation.

@forgetso because your comment has actionable information, I'll inform you that `Connection: keep-alive` is the default for HTTP/1.1. If you don't want keep-alive behaviour, you can specify `Connection: close` instead.