Raibek/requests-github-issues huggingface dataset:
Requests does not support session key logging

I would really like requests to support session key logging.  Firefox and Chrome both support logging the symmetric session key used to encrypt TLS traffic.  This makes it easy to analyze traffic with Wireshark.
Requests should write symmetric session keys to the file set in user variable “SSLKEYLOGFILE”.

Requests cannot do this in normal operation because OpenSSL does not expose appropriate APIs to extract this keying information. Both Chrome and Firefox use other TLS implementations (BoringSSL and NSS) which do have the appropriate APIs.

I am currently working on a patch to enable this functionality in openssl/openssl#1646. Please track that issue, as until it merges it will not be possible for Requests to support this function. It will also be limited to OpenSSL 1.1.0 and modern Python versions. 


Hi Cory, it seems your change has been merged to OpenSSL. Is the key logging supported in requests now?


Nope. Adding support for this requires first that OpenSSL 1.1.1 be released, and second that bindings for these functions be added to our dependencies. When OpenSSL 1.1.1 releases I'll add bindings to PyOpenSSL, but the stdlib will take a long time to catch up.

[OpenSSL 1.1.1 is out](https://www.openssl.org/blog/blog/2018/09/11/release111/)!
News on the dependencies front?

Better to ask pyOpenSSL if they support it yet. Then convince urllib3 to expose a way to utilize it from their pyOpenSSL shim. .... Then it will be actually useful to bug us about how to use it within requests.

Is this feature available yet in requests? It seems to still not be respecting the environment variable, but maybe I am just overlooking something? It would provide a very easy way to debug some of the requests in my script.

Also found this: https://bugs.python.org/issue34271

Just use https://pypi.org/project/sslkeylog/, it works for me.

@gtors 
I've tried the sslkeylog (0.2.0) module but with mixed success.  A line is written into the sslkeylog.log file for each request.{get/put} but wireshark only seems to recognize and use some of them.  For example, if my code has six requests.get() calls, six entries will be written to sslkeylog.log but wireshark will only be able to decode, say, three of the calls in a trace.

I suspect this is because 'requests' uses 'pyopenssl' but that's just a guess. This code is beyond my python knowledge.

If anyone has a suggestion, I would love to hear it.

Thanks
Typo in Quickstart Guide?

Reference: https://github.com/kennethreitz/requests/blob/master/docs/user/quickstart.rst#cookies
```
>>> jar.set('tasty_cookie', 'yum', site='httpbin.org', path='/cookies')
>>> jar.set('gross_cookie', 'blech', site='httpbin.org', path='/elsewhere')
```
Running this example I get the following error:
`TypeError: create_cookie() got unexpected keyword arguments: ['site']`

Shouldn't `site=` be `domain=`? The error goes away if I try that. If so I'll be glad to make a PR, I just want to make sure I'm not missing something

Good spot, @Cleod9! `domain` is definitely the keyword we want here. A PR for this would be very welcome.


No problem, I should be able to get a PR up within the next couple days


PR is up :)


Resolved by #3691.
Requests library becomes broken, forcing reinstallation solves the problem

It seems that I am not the only one that discovered that his python-requests library is in a broken state.
```
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 64, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 29, in <module>
    from .exceptions import InvalidURL, InvalidHeader, FileModeWarning
ImportError: cannot import name InvalidHeader
```

See https://bugzilla.redhat.com/show_bug.cgi?id=1393979 and http://stackoverflow.com/questions/32986626/python-requests-importerror-cannot-import-name-headerparsingerror which are two unrelated occurences that are fixed by the same workaround:

`pip install --upgrade --force-reinstall requestsexceptions requests requests-toolbelt`

#### pip freeze
I mention that the forced reinstallation did not change the reported versions but fixed the import error.
```
requests==2.11.1
requests-toolbelt==0.7.0
requestsexceptions==1.1.3
urllib3==1.15.1
```


This is unequivocally not a Requests issue. Requests defines `InvalidHeader` in the `exceptions` module, that import has to work. If it doesn't work, it's the result of something messing with the requests exceptions heirarchy. Given that you have installed a package called `requestsexceptions` you should strongly consider the possibility that _they_ broke your software, not us.


FYI, requestsexceptions looks like it doesn't monkey-patch `requests.exceptions`: https://github.com/openstack-infra/requestsexceptions/blob/master/requestsexceptions/__init__.py.

Either way, force reinstalling indicates that this is emphatically not our problem (as @Lukasa has already pointed out).


Also, @ssbarnea, http://stackoverflow.com/questions/32986626/python-requests-importerror-cannot-import-name-headerparsingerror seems to be a different (although similar) issue. From the StackOverflow issue it looks like you have some left over bits of old versions of requests on your disk.


For the lazy, the commands that successfully fixed this are:

```
sudo pip uninstall requests
sudo pip install --upgrade requests
```
If-Modified-Since is a "special" header?

I am using requests (2.10.0) in Python 2.7.11 and I just ran across some very strange behavior with respect to this header. I would like to send along different formats than 'Tue, 08 Nov 2016 05:00:00 GMT' which seems to be passed along just fine, but if I try a string like '2016-11-08T00:00:00-05:00' something is munging that value, fails to parse it (but does not raise an error) and my services on the other side get a completely different date (Thu, 31 Dec 2015 00:00:00 GMT).

So something is going on here, I'm just not sure where or even why. I am wondering if there is a way to simply pass along the string as-is...

@miscellainiac we do not (nor does anything between us and the socket) munge that header. My guess is that there's something infront of the server (e.g., a proxy or load balancer) that does this to your request header. Either way, the correct format is like your first example. See
- https://tools.ietf.org/html/rfc7231#section-5.2
- https://tools.ietf.org/html/rfc7232#section-3.3
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since

The first two are parts of the standard that define the behaviour and format of that header's value. The latter is a simplistic way of formatting it for ease.
json parameter added to patch and put methods

referring to #2979 ...

this is answered that this can be done with **kwargs, which is true. But wouldn't that would go for **post** too ?
It would make consistent use of parameters possible if **put** and **patch** would support the _json_ parameter also, so then there is no need for additional logic to distinguish between the different methods, take for example:

```python
   ...
   self.client = requests.Session()  
   ...
   request_args = {}
   if method == 'get':
        request_args['params'] = params
   elif hasattr(endpoint, "data") and endpoint.data:
       request_args['json'] = endpoint.data
   ....
   func = getattr(self.client, method)
   response = func(url, stream=stream, headers=headers, **request_args)
   ...

```
post, put and patch can all be handled the same this way.

>  if put and patch would support the json parameter also

They already do. 


Yup, you are right. 

Somehow reading the docs one gets a direction because the **post** method explicit names  the **json** parameter in it's documentation. While the **put** and **post** don't, they indirectly refer to the params of request.


@hootnot I think in this case this should be considered a documentation issue, rather than explicitly naming the parameters. Mind opening a docs PR?
[httpAdapter] allow empty password in proxy credentials

fixes #3659

Some things that could be improved in future:
- add some unit test using real proxy, e.g. MITM-proxy, something along the lines of Scrapy: https://github.com/scrapy/scrapy/blob/129421c7e31b89b9b0f9c5f7d8ae59e47df36091/tests/test_proxy_connect.py#L35 where there is fake proxy started and all tests for proxy requests go via this fake proxy. This would be nice to have to test somethings with real proxies (instead of doing unit test of some specific adapter method).


Cool, so this looks really good. You'll note we have some tests with proxies in `test_lowlevel.py`, so if you feel like adding more testing in this area I'll happily merge it. However, for this specific issue, I think this test is sufficient.

Thanks @pawelmhm! :sparkles: :cake: :sparkles:
Drop Python 2.6 support

Yesterday marks 3 years since the last release of Python 2.6! 🎉

To celebrate, I'm attempting to drop support for it from 156 prominent Python packages (one for every week it's past end-of-life)--including this one!

I've tried my best to remove as much 2.6-specific cruft as I can, but at the very least, this PR will remove the `'Programming Language :: Python :: 2.6'` trove classifier from this projects `setup.py`.


No.


Double no.


One year on, any thoughts on dropping 2.6?

If not now, what are the reasons for keeping it and when might it be droppable?

I see it's no longer being tested: https://github.com/requests/requests/commit/c3ecb825c41f90fb6bec28c282c70c84398e08c0.

This was eventually done in #4118.
Accept integers in header

This is a fix for the following crash of the utils.py I've encountered when syncing with vdirsyncer.

Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/vdirsyncer/cli/tasks.py", line 58, in sync_collection
    force_delete=force_delete
  File "/usr/lib/python3.5/site-packages/vdirsyncer/sync.py", line 169, in sync
    b_info.prepare_idents(storage_a.read_only)
  File "/usr/lib/python3.5/site-packages/vdirsyncer/sync.py", line 104, in prepare_idents
    for href, etag in self.storage.list():
  File "/usr/lib/python3.5/site-packages/vdirsyncer/storage/dav.py", line 565, in list
    headers=headers)
  File "/usr/lib/python3.5/site-packages/vdirsyncer/storage/dav.py", line 322, in request
    return utils.http.request(method, url, session=self._session, *_more)
  File "/usr/lib/python3.5/site-packages/vdirsyncer/utils/http.py", line 79, in request
    r = func(method, url, *_kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 462, in request
    prep = self.prepare_request(req)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 395, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 301, in prepare
    self.prepare_headers(headers)
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 415, in prepare_headers
    check_header_validity(header)
  File "/usr/lib/python3.5/site-packages/requests/utils.py", line 794, in check_header_validity
    "not %s" % (value, type(value)))
requests.exceptions.InvalidHeader: Header value 1 must be of type str or bytes, not <class 'int'>


Thanks! However, this is not considered a bug. See #3477 for more.
requests can't properly handle 'WWW-Authenticate' response header 

when the http response like this:
`HTTP/1.1 401 Unauthorized\r\n
Server: DNVRS-Webs\r\n
Content-Type: text/html\r\n
WWW-Authenticate: Basic realm="DVRNVRDVS"\r\n
WWW-Authenticate: Digest realm="DVRNVRDVS", domain="::", qop="auth", nonce="2c4158c9159934737e69aa698494ae04", opaque="", algorithm="MD5", stale="FALSE"\r\n`
I have print the response headers ,`{'Server': 'DNVRS-Webs', 'Content-Type': 'text/html', 'WWW-Authenticate': 'Basic realm="DVRNVRDVS", Digest realm="DVRNVRDVS", domain="::", qop="auth", nonce="2c4158c9159934737e69aa698494ae04", opaque="", algorithm="MD5", stale="FALSE"'}`,is this a bug? @nakajima @alex @mkomitee @thedaniel 


This isn't a bug in Requests. You're getting a 401, which means you're using or were provided the wrong credentials for the server you're attempting to authenticate to.


@drpoggi is likely correct. Further, without extra information, it's hard for any of the people you tagged in this bug to help you (let alone the maintainers of this project).

If you have a _question_ please ask it on [StackOverflow](https://stackoverflow.com).


I presume the _actual_ question is: is it a bug that the two WWW-Authenticate headers got merged into one.

The answer is no. WWW-Authenticate is defined as a list header, and the relevant RFC (RFC 7235 for those keeping track) explicitly points out that the header may contain multiple challenges, each of which is comma-separated, as well as the various comma-separated parameters. What requests did was totally fine here.


here is a example,`import requests
from requests.auth import *
req = requests.get('http://61.91.211.158:81/onvif/snapshot/101',timeout=3)
print req.headers`,but it raise timeout exception,I have used wireshark to capture the packets,It response quikly.`HTTP/1.1 401 Unauthorized
Date: Sat, 29 Oct 2016 19:06:59 GMT
Server: DNVRS-Webs
Cache-Control: no-cache
Content-Type: text/html
Connection: keep-alive
Keep-Alive: timeout=60, max=99
WWW-Authenticate: Basic realm="DVRNVRDVS"
WWW-Authenticate: Digest realm="DVRNVRDVS", domain="::", qop="auth", nonce="71225169d5e6c3e80672d195911b6d23", opaque="", algorithm="MD5", stale="FALSE"`,why requests will raise timeout exception? ,and when I remove timeout param It will get 401 status code,but will wait a long long time.@Lukasa @sigmavirus24 @drpoggi 


I suspect that the issue is that the response is not correctly terminated, which causes Requests to expect more data to show up. Is there any body content? Because having a Content-Type suggests there is.
Cannot import requests when filename is email.py

This is a weird error. But if you attempt to import requests from a filename `email.py` importing will fail. 
## Steps to Reproduce
1. `mkdir potato && cd potato`
2. `vim email.py`
3. `echo 'import requests' > email.py`
4. `python email.py`
## Actual Result (bug)

Instead of importing, it will start to ask for specific dependencies.

``` bash
$ python email.py
Traceback (most recent call last):
  File "email.py", line 1, in <module>
    import requests
  File "/Library/Python/2.7/site-packages/requests/__init__.py", line 60, in <module>
    from .packages.urllib3.exceptions import DependencyWarning
  File "/Library/Python/2.7/site-packages/requests/packages/__init__.py", line 29, in <module>
    import urllib3
ImportError: No module named urllib3
```
## Machine information

Using macOS default Python `v2.7.10` and default `pip` located at `/Library/Python/2.7/site-packages/requests`


`email` is a standard library package name, you should _never_ shadow those. We, rely on it not being shadowed and use it, which is why you cannot import requests. Please change your file name.
The data= parameter barfs on all dictionary-like objects that don't subclass dict itself

Example:

``` python
import requests
from collections import MutableMapping

class mydict(MutableMapping):
  def __init__(self, *args, **kwargs):
    self.d = dict(*args, **kwargs)
  def __iter__(self):
    return iter(self.d)
  def __getitem__(self, key):
    return self.d.get(key)
  def __setitem__(self, key, value):
    self.d[key] = value
  def __delitem__(self, key):
    del self.d[key]
  def __len__(self):
    return len(self.d)

d = {'key':'value'}
myd = mydict(d)

# Works.
print requests.post('http://httpbin.org/post', data=d).json().get('form')

# Should also work but doesn't.
print requests.post('http://httpbin.org/post', data=myd).json().get('form')
```

In models.py, the `is_stream` check

``` python
is_stream = all([
    hasattr(data, '__iter__'),
    not isinstance(data, (basestring, list, tuple, dict))
])
```

explicitly checks if the `data` parameter is an instance of `dict`, so only
instances of `dict` can be passed through `data=`. All other dictionary-like
objects later bomb with

``` python
TypeError: must be convertible to a buffer, not mydict
```

Is it intentional that all dictionary-like objects not explicitly `dict`
subclasses are rejected? I can't fathom why.


Nope, probably we should check for `Mapping` instead. Would you like to provide a PR to fix this up?


We had this at some point. Why did we get rid of this?


I'm honestly not sure. =(


Hello,
I am a seasoned Python developer and I want to make my first steps 
as an open source contributor. Can you assign this bug to me?


@gardiac2002 Unfortunately, GitHub doesn't allow assigning issues to anyone other than contributors or the person who opened the issue. However, no-one on the team is currently working on this, so you are welcome to work on it and open a pull request.


@Lukasa thank you, I am going to take a look at the issue :)


I created a pull request for the issue: https://github.com/kennethreitz/requests/pull/3652
All tests ran successful from Python2.6 to Python3.5


It looks like this is now resolved thanks to @gardiac2002's work in #3652.
malformed request block

specifying both Content-Length and transfer-encoding in the header dictionary creates malformed request block. I think it should throw an exception and abort the request


I'm going to close this in favour of #3637.


This is a different issue by the way . Code to reproduce:

`
import gevent
import socket
import requests
import os
from threading import Thread
import sys
bSTOP = False
def handleResponse(response,_args, *_kwargs):
    print(response.status_code)

def gen():
    yield 'pforpersia,champaignurbana'.encode('utf-8')
    yield 'there'.encode('utf-8')

def txn_replay():
    try:
        request_session = requests.Session()
        hostname = "127.0.0.1"
        port = "8080"
        request_session.proxies =  {"http": "http://{0}:{1}".format(hostname, port)}
        hdr = {'Host':'www.blabla.com','content-type': 'application/json', 'Content-Length':'20' 
, 'Content-MD5':'5f4308e950ab4d7188e96ddf740855ec', 'Transfer-Encoding':'Chunked'}
#, 'Content-Length':'20'
        body=gen()
        response = request_session.get('http://blabla.com/blabla', headers=hdr, stream=True, data=body)
    except UnicodeEncodeError as e:
        print("UnicodeEncodeError exception")

```
except requests.exceptions.ContentDecodingError as e:
    print("ContentDecodingError",e)
except:
    e=sys.exc_info()
    print("ERROR in requests: ",e)
```

def main():
    txn_replay()

if **name** == '**main**':
    main()
`


**wireshark**

`GET http://blabla.com/blabla HTTP/1.1
Accept: _/_
Connection: keep-alive
User-Agent: python-requests/2.10.0
Accept-Encoding: gzip, deflate
content-type: application/json
Content-Length: 20
Host: www.blabla.com
Content-MD5: 5f4308e950ab4d7188e96ddf740855ec
Transfer-Encoding: chunked

pforpersia,champaignurbanatherethere

HTTP/1.1 502 Success
Date: Tue, 25 Oct 2016 20:41:08 GMT
Connection: close
Server: ATS/7.1.0
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 247`


The problem is that Session generates the request block which is basically malformed with transfer-encoding and content-length being specified and adding message body. This should either take the content-length field out or raise an exception 


This behaviour has been known about Requests for a long time and is intentional. Requests basically totally disregards the user-set headers that have anything to do with request framing (content-length or transfer-encoding). You must not set them.
Update SSL options on proxy pool manager.

In a previous PR, the SSL options are updated on the main pool manager in `get_connection()` but not when the selected pool manager is a proxy pool manager.  This PR fixes that.  

In addition, I renamed the `_pool_kw_lock` because it's used in  `get_connection()` to create a _critical section_ for both updating the selected pool manager _connection_pool_kw_  and calling `connection_from_url()` without being preempted.


Thanks for this @jortel!

I think that @jeremycline is planning a related PR that focuses on changes in urllib3 first. @jeremycline, how does this relate to your plan and what are your thoughts about this approach versus yours?


Hey @jortel, I filed #3633 and attached a few proposed patches. Let me know what you think. I need to take the time to write a few tests before I submit a PR, but I will probably have time for that tomorrow night. 

I like making a change to urllib3 first because that way the issue of thread safety is all bundled into urllib3 rather than the responsibility of each user of urllib3. 


@jeremycline I like what you've proposed in #3633 and will close this PR in favor of that effort pending the outcome.
Keep ``verify`` setting when no CA_BUNDLE variable exists

If the `trust_env` flag is set on a session and `verify` is `True`
or `None`, the environment is checked for `CURL_CA_BUNDLE` and
`REQUESTS_CA_BUNDLE`. Before this patch, if neither existed,
`verify` would always be set to `None` rather than `True` when it
was originally `True`.

I found this while working on a patch for https://github.com/kennethreitz/requests/issues/3633.


@Lukasa I've adjusted the test.


@Lukasa no worries. I haven't really worked with pytest and didn't see examples of patching in the existing tests so I got lazy :frowning_face:.


Hurrah! No need to worry @jeremycline, this is what code review is for! I'm delighted to say that I'm now happy with this patch, and ready to merge. Thanks so much! :sparkles: :cake: :sparkles:
Inconsistent results when querying a server with requests.post(data=dict)

Python version: 2.7.12 and 3.5.2
Requests version: 2.11.1

I'm sorry if this is a duplicate, but I don't know what the issue is, so I also don't even know how to look for it in the search... But this might be a bug, so I decided to report.

When I run this in Python 2:

``` python
import requests


response = requests.post('http://evds.tcmb.gov.tr/cgi-bin/famecgi', data={
    'cgi': '$ozetweb',
    'ARAVERIGRUP': 'bie_yymkpyuk.db',
    'DIL': 'UK',
    'ONDALIK': '5',
    'wfmultiple_selection': 'ZAMANSERILERI',
    'f_begdt': '07-01-2005',
    'f_enddt': '07-10-2016',
    'ZAMANSERILERI': ['TP.PYUK1', 'TP.PYUK2', 'TP.PYUK21', 'TP.PYUK22', 'TP.PYUK3', 'TP.PYUK4', 'TP.PYUK5', 'TP.PYUK6'],
    'YON': '3',
    'SUBMITDEG': 'Report',
    'GRTYPE': '1',
    'EPOSTA': 'xxx',
    'RESIMPOSTA': '***',
})

print(response.text)
```

I always get the same response, consistently, no matter how many times. If I run this in Python 3, I get a different response. I checked the request body being sent to the server, and the request headers, and they are all the same in both versions of Python. The correct response is obtained from running this in Python 2.

Since both versions of Python are supported, and the same API is exposed, then the same result should happen. I also asked this in Stack Overflow, and at least one other developer was able to reprodeuce and confirm this happens: http://stackoverflow.com/questions/40118133/library-requests-getting-different-results-unpredictably

You should check the discussion at Stack Overflow, as some people entertained some more scenarios.


The server cares about the order of the parameters. Closing the issue as it is invalid.
remove RequestsCookieJar specific update call

This is a minor fix in the same vein as #3591. This function calls `update` on the CookieJar which only exists on `RequestsCookieJar` not the standard library `cookielib.CookieJar`. While this function will now be somewhat trivial, this will ensure it maintains backwards compatibility. It may be worth discussing removal in the future.

I have a test as well @Lukasa but it seems a bit overkill. I can include it if you'd like though.


Hey @Lukasa, just pinging on this when you've got a moment. This should be the last problematic use of `update` on CookieJars in the codebase.


Thanks @nateprewitt! :sparkles:
Use merge_environment_settings method in sessions.send method

I work on a python project that uses requests. And as a developer, I have set that enviroment variables: HTTP_PROXY, HTTPS_PROXY and REQUESTS_CA_BUNDLE so I can see all of my requests with mitmproxy. Today, I have decided to use session.send() method with a prepared request instead of using .get(), .post() or .request(). But then, I realize that .send() doesn't respect my environment variables.

So, I think that merge_environment_settings method should be used in .send() method, instead of .request(). What do you think about that? Should I open a pull request for that?


@yozel Thanks for this issue!

Per #2807, this is intentional behaviour. `merge_environment_settings` is a public, documented method on the `Session`: if you're planning to move away from using `Session.request` to the prepared request flow, this is one of the things that needs to become part of your workflow if you want its behaviour.
fix issue when the file-like object raises an IOError with tell


Good spot!

Unfortunately, this fix has the effect of ensuring that files of this type pass transparently but send no data, as we've called `seek` on them to seek to the end. In this instance, probably the easiest fix is to move the `if` block to an `else` on the original `try`. That way, we know that the previous call to `tell()` didn't explode.


Isn't is possible for seek itself to give an IOError?

for example:

``` python
import os
f = os.fdopen(0) # getting a file object for  the standard input
f.seek(10)
```

will give an IOError.

So I wonder if they can be handled with the same `except`, or they should be handled separately.


@mie00 The question isn't "can `seek()` throw an `IOError`", it's "can `seek()` throw an `IOError` in a situation where `tell()` does not". I feel like the odds of that are pretty low. I'm inclined to want to worry about that when we know it's an issue.


I tried to find a case, but I couldn't. The only thing that I was able to find is that `tell` giving `IOError` while `seek` gives `UnsupportedOperation` which is handled elsewhere (for example `requests.Response.raw`).

So, PR updated.


Ok, rather than block on @sigmavirus24 having time to swing back to this, I'll merge this now. @sigmavirus24 can provide feedback whenever he gets time back. Thanks for the work @mie00!
how can I pass gmail's cookie to requests session?

Hello
assume that i define a session
`r = requests.Session`
and i want to set cookie for it : 
`r.cookies = my_cookies`
I know that I must use cookie jar . and I know taht I must use dictionary format. But I have a problem!
normally cookies are in this format
`ukey=*****; __qca=*****; _ga=******;`
and they are convertable to doctionary format easily
`{'ukey' : '****' , '__qca' : '****' , '__ga' : '*****' }`
but how can deal whith gmail's cookie ? they have diffrent format , for example
`S=gmail=****; COMPASS=gmail=****;`
I tested SimpleCookie , that some one requested in this answer at stackoverflow.com
http://stackoverflow.com/questions/32281041/converting-cookie-string-into-python-dict
but I got this error message
`File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 510, in head
    return self.request('HEAD', url, **kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.5/site-packages/requests/sessions.py", line 611, in send
    extract_cookies_to_jar(self.cookies, request, r.raw)
  File "/usr/lib/python3.5/site-packages/requests/cookies.py", line 133, in extract_cookies_to_jar
    jar.extract_cookies(res, req)
AttributeError: 'dict' object has no attribute 'extract_cookies'
`
Please guid me


Hey @alireza-amirsamimi, thanks for opening this ticket. Requests currently doesn't support passing a `dict` object to Session's `cookies` attribute. You must first convert your dictionary into a `CookieJar`-like object before passing it to the session. I've attached the code below that you'll need to solve your issue. In the future, it's preferred that questions regarding how to use Requests are opened on Stackoverflow instead of the Github ticket tracker. Thanks again!

e.g.

``` python
from http.cookies import SimpleCookie
from requests.cookies import cookiejar_from_dict
from requests import Session

r = Session()
my_cookie = SimpleCookie()
my_cookie.load('rawcookie=string;')

cookies = {key: morsel.value for key, morsel in my_cookie.items()}

r.cookies = cookiejar_from_dict(cookies)
```

Just to make note of it, this is an occurrence of #3595.
Preparing POST request hook 

First, as countless have said, I'd like to thank all of the contributors for such an excellent package.

I am trying to maintain an auto-incrementing id for a given session and url every time I submit a post request. For example, say I have an authenticated session object.
### Example

``` python
ID_KEY = 'id'
counter = 1
data = 'blah'
payload = {ID_KEY: counter, 'other_stuff': data}
resp = session.post(SPECIFIC_URL, json=payload)
counter += 1
payload = {ID_KEY: counter, 'other_stuff': data}
resp2 = session.post(SPECIFIC_URL, json=payload)
```

My easy solution is having an adapter class that wraps the session object with its own post method, but I was wondering if I could use either a prepared request or hook that would achieve the same effect more cleanly?


A Session wrapper or something similar is the best approach here. We got rid of most of our hooks, and you can't hook into the request factory easily.
Update makefile to only ship tagged releases.

Extra work that spun out of #3620.

We've had it as our policy that we only ship tagged releases of our vendored modules for a while now. This makefile update consolidates that by ensuring that the makefile will only ever use the latest tag for updates.

For the moment this is probably a bit naive, because if any of our dependencies tagged backport releases we'd be totally screwed. Right now they don't, and I think we'll notice if they start doing it, so we should be safe.
Better support for internationalized domain names.

Fixes #3616. This adds support for IDNA 2008 by vendoring the idna module, with the kind permission of @kjd.

For those keeping track, changes like this are another reason that Requests should stay out of the stdlib. ;) See also: [CPython issue 17305](https://bugs.python.org/issue17305).


"This adds support for IDNA 2008 by vendoring the idna module"

Why copying the whole module? Why not using a dependency? requests already uses requirements.txt.


@haypo Without wanting to get too far into the weeds here, it remains the policy of the Requests project that we vendor all dependencies. If you'd like to discuss the amendment of that policy I recommend you email Kenneth, as all such decisions are his to make.

You'll also note that our requirements.txt are _development_ requirements, not our actual requirements for functioning.


"it remains the policy of the Requests project that we vendor all dependencies"

Oh ok. I misunderstood the requirements.txt file.


@haypo No problem. =) This is an unusual policy, and it'll be interesting to see how much longer we keep it, but on a personal level I'd be totally happy to unvendor our deps.


Ok, addressed that issue.


Cool, nothing else jumps out at me :shipit:


I'll change the way imports are performed in the upstream.


:cake:
ValueError: negative shift count in _proxy_bypass_macosx_sysconf

Pls see https://github.com/pypa/pip/issues/4009


Sorry you were referred here, but this isn't a requests problem _either_. You should be able to reproduce the problem with this code, which uses entirely modules in the standard library:

``` python
from urllib.request import proxy_bypass
from urllib.parse import urlparse

netloc = urlparse("https://pypi.python.org/pypi").netloc
proxy_bypass(netloc)
```

I should note that I'm a little bit confused here as to why you're encountering this problem with the config you showed in the other post. Is your HTTPS config the same?


My HTTPS config is absolutely the same.


Were you able to verify the problem with the standard library code?


Closing due to inactivity.


Sorry for the delated answer
I cannot recreate the problem with the lib code, but the network config is changed a bit. (No major changes though) 


@AndiCui So, to be clear, using Requests gives you the problem but using the code above does not.


I retried once. This time it recreated the issue.
The problem happens when I have a “Location” enabled that is other than the Location which I had that config. 

```
>>> proxy_bypass(netloc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 2546, in proxy_bypass
    return proxy_bypass_macosx_sysconf(host)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 2523, in proxy_bypass_macosx_sysconf
    return _proxy_bypass_macosx_sysconf(host, proxy_settings)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/urllib/request.py", line 2509, in _proxy_bypass_macosx_sysconf
    if (hostIP >> mask) == (base >> mask):
ValueError: negative shift count
```


So this is not a Requests issue I'm afraid. 😔


To be clear @AndiCui your problem is arising from the standard library, not from within Requests itself. If you're seeing this problem using Requests, we can handle it, but we the ValueError being raised needs to be addressed in the standard library and you should file a bug on bugs.python.org for that.
form action without schema'ed ACTION fails with MissingSchema

Used `requests` (and mechanicalsoup) to .get() a web page that has a form on it. The form tag looks like

`<form action="/submit_uri" method="post">`

I filled a part of the form, and then .submit()ed the form.

```
  File mechanicalsoup/browser.py", line 114, in submit
    request = self._prepare_request(form, url, **kwargs)
  File "mechanicalsoup/browser.py", line 109, in _prepare_request
    return self.session.prepare_request(request)
  File "requests/sessions.py", line 394, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "requests/models.py", line 294, in prepare
    self.prepare_url(url, params)
  File "requests/models.py", line 354, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '/submit_uri': No schema supplied. Perhaps you meant http:///submit_uri?
```


You're telling Requests to `get` at the URI of `/submit_uri` which isn't a valid URI.

You need to get the URI of the form, IE, `https://www.example.com` and add this to the URI before sending the `get` request.


Correct.

Resolving a URL you find in the HTML of a page into a gettable URL is an extremely complex operation, and Requests doesn't do it for you because it's fundamentally not a HTML library. The [WHATWG URL Specification](https://url.spec.whatwg.org) defines the logic for taking a base URL (the URL of the page you requested) and processing a URL on that page (in this case a path-absolute URL) into a new URL. Unfortunately, I am aware of no Python library that fully implements that logic at this time, but something like [rfc3986](http://pypi.python.org/pypi/rfc3986) is probably good enough.


Here's what I expect out of requests:

Methods that request pages newly take a context url, and interpret the passed in destination, pulling from the context url when necessary.

`get("foo", context="http://other/location/bar")`  gets `http://other/location/foo`

`get("/baz", context="http://other/location/bar")` gets `http://other/baz`

`get("://third/baz", context="https://other/location/bar")` gets `https://third/baz`


Why would you expect that out of Requests? Building URLs in this way is not a HTTP concern.


@chadmiller other libraries will do that for you (e.g., `rfc3986`) because they are not in the HTTP specification, they're in other (URI) specifications.
Consider vendoring the IDNA library.

As has been noted on the [Python Security SIG mailing list](https://mail.python.org/pipermail/security-sig/2016-October/000122.html), Python currently only includes the deprecated IDNA 2003 codec in the standard library. This is problematic, because IDNA 2003 is forbidden for some ccTLDs, such as `.de`: instead, the newer IDNA 2008 standard should be used. That standard is implemented in the PyPI package `idna`.

If we're going to support IDNA 2008, I would like it to not be optional: it leads to a fairly substantial and difficult to debug change, so it should be supported fully. To do that and keep in touch with Kenneth's wishes regarding dependencies, we'd have to vendor it.

I'd like contributors and packagers, particularly @eriol, @ralphbean, and @sigmavirus24 to weigh in with their thoughts here. What are your thoughts here? I'd also, if possible, like to hear from @kjd: while `idna` is licensed under a BSD-like license that should be broadly compatible with our own Apache 2.0 and so should present no legal blockers to vendoring, I'd still rather do it with the original author's permission than without it.


It is fine by me. Happy to support widespread use of the modern IDNA standard however I can.


I'm 👍 on it. I think making it non-optional is an excellent idea.


+1 for me. I agree with @sigmavirus24 on making it non-optional.


Hey, @ralphbean has asked me to take over being the point of contact for Fedora packaging interactions.

I'm fine with this and I'm also fine with it being non-optional.


Thanks for your feedback folks! I've done the work in #3620: please cast your eye over it. This is particularly true for downstream packagers, who will likely want to consider the effect this dependency will have on your packaging ecosystems.
Support 100-continue (expect100)

This feature is used when client wants to check whether request body will be accepted by server before sending the body.

This is achieved by setting "Expect: 100-continue" request header. Server will respond with status line "100 Continue HTTP/1.1" if request headers are ok, or with actual full response if something goes wrong. Client should not send body if response was not "100 Continue HTTP/1.1"

Some useful links:

Related section in pep333:
 https://www.python.org/dev/peps/pep-0333/#http-1-1-expect-continue

How it looks like in aiohttp client (see expect100 parameter):
 http://aiohttp.readthedocs.io/en/stable/client_reference.html#basic-api

More detailed description:
 https://tools.ietf.org/html/rfc7231#section-5.1.1


Hi @redixin ,

After a quick search I found https://github.com/kennethreitz/requests/issues/713 which is an old issue that provides the reasoning for why Requests does not support this flow. In fact, this is because we are built on top of httplib. If you want to support 100-continue properly in Requests, you need to go convince the core python developers to support this first.

In the future, please search for similar issues.
Update streaming request headers

Is there any way to update headers on a streaming request without restarting the request? 

I am using an Authorization header with tokens that go stale after an hour, so without updating the header to include new tokens a stream can only run for an hour.


Headers are sent at the start of the message. How do you think you would update them without a new request?

The RFC specifies "trailers" but those are for the end of the message, not for the middle of a message and those aren't widely supported (and thus are not supported by requests).
Memory Bug in requests when being used with subprocess module.

So I ran into this while working on a more complex project however here is a short demo to showcase the bug.
**NOTE:** I highly recommend you open some type of task viewer program to monitor RAM usage and know when to `Ctrl C` to prevent your OS from crashing.
# Steps to Reproduce:

a. Create Directory
b. Create two new files in the directory called script.py and queue.py
c. Put the following code into script.py:

```
import requests
```

d. Put the following code into queue.py:

```
from subprocess import call
call(['python','script.py'])
```

e. Now run the following command:

```
python script.py
```

Your computer should now begin rapidly using RAM and will eventually crash (took about 30 seconds for my Ubuntu install to crash on a rig with 8 GB of RAM.)


So this is happening because of how `six`, a package used by Requests, does its module discovery. You can confirm by changing `import requests` to `import six.moves.queue` (the line in `urllib3` causing this) and you'll get the same behavior. Your queue.py ends up getting imported and run, which obviously results in an infinite loop and your PC spawns Python processes until it crashes.

The solution is to name queue.py something else, preferably not a name shadowing a standard library module, because as you can see that can result in some very weird behavior and is generally considered a bad idea. If you want to file a bug with six their repo is [here](https://bitbucket.org/gutworth/six) but the simpler solution is probably you just change your filename.


@drpoggi is correct. There's not much to be done here within requests (or six for that matter). It's usually best to not to shadow the names of standard library modules.
MKCOL verb from WebDAV

MKCOL would be super useful - make a directory on the remote server.

curl has it:

```
curl -u user:pwd -X MKCOL http://yourserver.com/baseurl/newdiruwannacreate 
```


From what I know the -X flag means you're using a custom metbod, laying
outside normal HTTP requests.

You should be able to achieve this using `requests.request('MKCOL',
**kwargs)`

I may be wrong though.

On Thu, 6 Oct 2016, 8:29 PM Paul Hammant notifications@github.com wrote:

> MKCOL would be super useful - make a directory on the remote server.
> 
> curl has it:
> 
> curl -u user:pwd -X MKCOL http://yourserver.com/baseurl/newdiruwannacreate
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/3611, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AGk_7Wb4lCaJIKH_tBRks0B4TB0DuM1lks5qxM2lgaJpZM4KPyV0
> .


@TetraEtc is correct. Cheers!


Thanks :)
Remove error swallowing exception catching of AttributeError

This exception catching is far too aggressive in my opinion. A lot of code could run in this and could throw AttributeError for any number of reasons. It's actually not clear to me what error it's trying to catch. It made it quite difficult for me to debug this problem: https://github.com/shazow/urllib3/pull/990

This PR removes it altogether since as far as I can see it is not needed in normal operation and can only serve to mask errors.


I did some ~~dumpster~~ history diving and found https://github.com/kennethreitz/requests/commit/f7968b6797a50716a2f177ad23439a3c6bb9f683 and https://github.com/kennethreitz/requests/commit/296d8cc097b694ec8a9c4d8d7c00abed4755e5d5

The first is linked to https://github.com/kennethreitz/requests/issues/236 which indicates that at times `Response.raw` used to possibly be None. I don't think this could happen any longer (I think 98% of the time there's a valid urllib3 HTTPResponse) and we no longer access `self.raw` directly in this method, so I suspect this is probably safe.
Link on Documentation web page not working

link for _connection pooling_ on [this](http://docs.python-requests.org/en/master/user/advanced/#session-objects), line 3 looks broken and shows message

_SORRY
This Page doesn't exist yet._


Seems that the old documentation on connection pooling moved around in urllib. The page it references (http://urllib3.readthedocs.io/en/1.5/pools.html) no longer exists after some refactoring was done.

Perhaps a better alternative is:
http://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool
or
http://urllib3.readthedocs.io/en/latest/advanced-usage.html#customizing-pool-behavior


Yup, that looks wrong. I recommend the first link you posted @bbamsch. Would either of you be interested in submitting a pull request to fix that?


I am working on it.


@Lukasa, If I understand right, first link you are referring to is http://urllib3.readthedocs.io/en/1.5/pools.html ?


No, sorry, I meant http://urllib3.readthedocs.io/en/latest/reference/index.html#module-urllib3.connectionpool


thank you


This is resolved with #3604.
when will the requests supports HTTP/2?

Our project need a http/2 client.


There is no roadmap for this. HTTP/2 is a complex protocol and it is not likely that Requests in its current form will achieve good support for it any time soon. In the meantime, you can [use this](https://hyper.readthedocs.io/en/latest/quickstart.html#requests-integration).


Read the docs appears to be down. Fyi

On Tue, 27 Sep 2016, 7:51 PM Cory Benfield notifications@github.com wrote:

> There is no roadmap for this. HTTP/2 is a complex protocol and it is not
> likely that Requests in its current form will achieve good support for it
> any time soon. In the meantime, you can use this
> https://hyper.readthedocs.io/en/latest/quickstart.html#requests-integration
> .
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/3601#issuecomment-249818864,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AGk_7ehXc8joUXLgtfbtMW3v9m-iQRUVks5quOb5gaJpZM4KHaKv
> .


Loading fine for me here.


All resolved. Must have been a temporary DNS issue.

On Tue, 27 Sep 2016, 7:58 PM Cory Benfield notifications@github.com wrote:

> Loading fine for me here.
> 
> —
> You are receiving this because you commented.
> 
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/3601#issuecomment-249820462,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AGk_7SXTxilVQFhiFn25z9gq-DKVx9qnks5quOiggaJpZM4KHaKv
> .
Issue #3597 - Avoid bytestring/str hodgepodge when resolving cookie from Request URL

When resolving the request full URL, the Host cookie is used to override the URL's host, in Python 3 this causes a mix of str and bytestring objects when the Host cookie is set to a bytestring. Patch checks that ParsedResult is str type and host is non-str type before attempting to decode bytestring to str.


Patch should address Issue #3597


All appears to be in order 😃 

```
============================= test session starts =============================
platform win32 -- Python 3.5.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: C:\Users\Brian\PycharmProjects\requests, inifile:
plugins: httpbin-0.0.7, cov-2.2.1, mock-0.11.0
collected 381 items

tests\test_hooks.py ...
tests\test_lowlevel.py .........
tests\test_requests.py .............................................................................................................................................X.........X....................................................
tests\test_structures.py ....................
tests\test_testserver.py ...........
tests\test_utils.py ..s...................................................................................................................................

============== 378 passed, 1 skipped, 2 xpassed in 55.53 seconds ==============
```


Went through your comments and reworked the module of internal references to to_native_string. 👍