Raibek/requests-github-issues huggingface dataset:
about Custom Headers

import requests
UA=“Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Maxthon; .NET CLR 1.1.4322; .NET CLR 2.0.50727)”
Fref = file("C:/ref.txt", 'r')  
dat1 = Fref.readlines()
Fref.close()
ref= random.choice(dat1)
headers = {
                    'User-Agent': UA,
                    'Accept': '_/_',
                    'Accept-Charset': 'ISO-8859-1,utf-8',
                    'Accept-Encoding': 'gzip,deflate',
                    'Accept-Language': 'en-us,en',
                    'Connection': 'keep-alive',
                    'Referer':ref     }
r = requests.get('http://httpbin.org/get', timeout=30, headers=headers)
r.content

---
## Result

{
  "url": "http://httpbin.org/get", 
  "headers": {
    "Content-Length": "", 
    "Accept-Encoding": "identity, deflate, compress, gzip", 
    "Accept": "_/_", 
    "Host": "httpbin.org", 
    "Referer": "http://www.python.org/", 
    "Content-Type": ""
  }, 
  "args": {}, 
  "origin": "10.108.62.241"
}

sorry,I am not good at english. when run with  headers like this, the page say it haven't User-Agent . when removed Referer,it is ok .any ideas?
Update:
F0=file("C:/ref.txt", 'r').readlines()
# print result is    ['http://www.python.org/\n', 'http://www.google.com/\n']

ref1=map(lambda x: x.strip(), F0)
ref = random.choice(ref1) 

then, it  is fine.


Everything should be fine, but there's no reason to add all of those headers. A big chunk of them are already included.

Try this

``` python
import requests

agent = 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; Maxthon; .NET CLR 1.1.4322; .NET CLR 2.0.50727)'
headers = {'User-Agent': agent}

r = requests.get('http://httpbin.org/get, headers=headers)
```
If requests.async has been removed, the docs should point people to the right place.

If this [comment is true](https://github.com/kennethreitz/requests/issues/239#issuecomment-6180202) and requests.async has been removed and is now GRequests, then we want to make sure that people are being pointed in the right place :). 

If this isn't the case then disregard this request.

Thank you for all your work @kennethreitz


Thanks! I'll try to get a documentation site up for grequests soon.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1562994) (merged c589d8a2 into c0ae3dd3).
Requests breaks on certain urls that use : in the query string

> > > import requests
> > > requests.get('http://online.wsj.com?CALL_URL=http://online.wsj.com/article/SB10001424052702303640104577436251166644714.html%3fmod=googlenews_wsj')
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/api.py", line 51, in get
> > >     return request('get', url, *_kwargs)
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/api.py", line 39, in request
> > >     return s.request(method=method, url=url, *_kwargs)
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/sessions.py", line 200, in request
> > >     r.send(prefetch=prefetch)
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/models.py", line 463, in send
> > >     conn = self._poolmanager.connection_from_url(url)
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/packages/urllib3/poolmanager.py", line 89, in connection_from_url
> > >     scheme, host, port = get_host(url)
> > >   File "/Users/lsemel/www/virtualenvs/muckrack/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 557, in get_host
> > >     port = int(port)
> > > ValueError: invalid literal for int() with base 10: ''


https://github.com/kennethreitz/requests/blob/develop/requests/packages/urllib3/util.py#L75 should probably chop off the query string before checking for a :


/me is embarrassed.

Could you open this bug under urllib3 please?


No problem, here you go: https://github.com/shazow/urllib3/issues/73


This is fixed in shazow/urllib3@1c22dec48bc57199d71b86476dbed8f027877824, feel free to pull @kennethreitz.

Also worth noting that this bug is partly due to the improperly encoded param. It does not manifest if we do this instead:

``` python
import requests
requests.get('http://online.wsj.com', params={'CALL_URL', 'http://online.wsj.com/article/SB10001424052702303640104577436251166644714.html%3fmod=googlenews_wsj'})
```


Thanks @shazow @kennethreitz!
Progress object/callbacks

It would be nice to have a progress object/callback available, similar to urlgrabber's progress_obj.  This would allow, for example, progress logging on a large PUT request.


Ideally you'd be able to just provide a generator for the contents of the stream, and you could track progress there.


An example of download/upload progress callbacks, from Pycurl: http://pycurl.sourceforge.net/doc/callbacks.html (PROGRESSFUNCTION option). Not the greatest API, but relatively simple. 
I'm using pycurl on a project only to avoid monkey-patching urlblib2 to track the upload of large media files over multipart uploads.
Checking text on a HEAD verb Request breaks!

When trying to check whether I can override the standard header 'Content-Type', I encountered what I consider to be a bug when I tried to do this:

```
   >>> requests.__version__
    '0.12.1'
    >>> res = requests.head('http://httpbin.org/headers')
    >>> res.headers
    {'content-length': '349', 'via': '1.0 mail.ds.co.ug (squid/3.1.6)', 'x-cache': 'MISS from mail.ds.co.ug', 'x-cache-lookup': 'MISS from mail.ds.co.ug:
    3128', 'server': 'gunicorn/0.13.4', 'connection': 'close', 'date': 'Thu, 07 Jun 2012 12:15:19 GMT', 'content-type': 'application/json'}
    >>> res.text
    Traceback (most recent call last):
      File "<input>", line 1, in <module>
      File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 809, in text
        content = str(self.content, encoding, errors='replace')
    TypeError: unicode() argument 2 must be string, not None
```

I think the favorable way (if at all the HEAD verb shouldn't provide a body -- which I think it doesn't), would be to set text to None. Am not yet sure whether this is a bug or feature, the docs on the HEAD verb don't clarify on what to expect of 'text' in this case.

Anyway, my intended usage seems to be this:

```
    >>> requests.__version__
    '0.12.1'
    >>> res = requests.get('http://httpbin.org/headers')
    >>> res.text
    u'{\n  "headers": {\n    "Content-Length": "", \n    "Via": "1.1 mail.ds.co.ug (squid/3.1.6)", \n    "Accept-Encoding": "identity, deflate, compress,
     gzip", \n    "Host": "httpbin.org", \n    "Accept": "*/*", \n    "User-Agent": "python-requests/0.12.1", \n    "Connection": "keep-alive", \n    "Ca
    che-Control": "max-age=2419200", \n    "Content-Type": ""\n  }\n}'
```


Looks like we should be checking if `response.content is None` before we do the conversion


+1 also the same for `res.json`


Fixed!


:cake: Awesome!


%-) nice
Allow passing a file or file-like object as data.

The underlying httplib already allows passing an open file object as body
to its HTTPConnection.request method. I think requests should allow that
as well.

This also fixes issue #292 in requests. httplib will automatically add the content-length header if a real file object is passed. This will most likely break for non-file streams but it is a start.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1555287) (merged f0e327f9 into c0ae3dd3).


This is a great change, thanks!

In the future, I plan to support streaming uploads with generators.


:sparkles: :cake: :sparkles:


:thumbsup: Thanks for accepting and for adding me to the AUTHORS file for the few lines I contributed ;-)
SSL module cannot handle unicode filenames

Here is the related bug report on [bugs.python.org](http://bugs.python.org/issue14907)

This problem will produce the following error:

```
>>> requests.get('https://google.com', cert=u'öäü.pem')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/api.py", line 54, in get
    return request('get', url, **kwargs)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/safe_mode.py", line 37, in wrapped
    return function(method, url, **kwargs)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/api.py", line 42, in request
    return s.request(method=method, url=url, **kwargs)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/sessions.py", line 230, in request
    r.send(prefetch=prefetch)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/models.py", line 602, in send
    timeout=self.timeout,
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 384, in urlopen
    body=body, headers=headers)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 254, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/httplib.py", line 992, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/httplib.py", line 814, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/httplib.py", line 776, in send
    self.connect()
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 100, in connect
    ca_certs=self.ca_certs)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/ssl.py", line 381, in wrap_socket
    ciphers=ciphers)
  File "/usr/local/Cellar/python/2.7.3/lib/python2.7/ssl.py", line 141, in __init__
    ciphers)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
```


I this this is a `wont-fix` for now. I suspect this will work fine in Python3, for obvious reasons :)


But it does not work for Python 2.x. Why is it closed?


@kennethreitz ?


Because you're not expected to pass unicode strings to most standard libraries in Py2 (unless explicitly stated so).


@simukis Well, the user is passing the string to `requests` and **not** to a standard library.


what @simukis said
response.json: document ALL the awesome features!

I just saw `response.json` in your #djangocon presentation, which is awesome and was still unknown to me. I updated the docs accordingly.

In case you don't like the 2nd commit, just ignore or revert it.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1548245) (merged f968fe01 into aa423183).


Thanks a ton :)


:sparkles: :cake: :sparkles:
Async

I just realized that you removed `requests.async`in Commit 32327f8f7c2f457d38130b5df664ec8f29cf3053. What are the reasons for it?

In case it's not going back, the async section needs to be removed from the docs: http://docs.python-requests.org/en/latest/user/advanced/#asynchronous-requests


It does indeed :)

The `async` submodule was removed to keep requests sharp and focused. 

The code was moved to a new module, called `grequests`:

https://github.com/kennethreitz/grequests

Documentation needs to updated for this.


the doc still has an empty section labelled 'async'.


Documentation needs to updated for this. :)


This is still confusing, and there are many links to the old page for anyone searching for "requests async", for example

http://stackoverflow.com/a/9522864/329700

Might want to keep the header/tag, and then link people to the other library, so they don't wonder what happened to it.
requests.get() ignores charset=UTF-8 and BOM 

I've seen some discussion that charset=UTF-8 isn't sufficient because requests looks at the http headers rather than the body of the request (and I can understand the desire to avoid parsing the html looking for the META tag).  However, if a page has a UTF-8 Byte Order Mark, I'd think that would be sufficient indication to treat the body as UTF-8.


If the server doesn't specify an encoding, one will automatically be detected via [chardet](http://pypi.python.org/pypi/chardet)


If you're saying that _requests_ uses _chardet_, then this isn't working properly for me. Perhaps I need to dive into the code to see what's happening?


Requests uses chardet if it is available, yes.

Can you share the URL you're having trouble with?


http://www.stanford.edu/dept/visitorinfo/activities/dining.html


Ah, this is because the server responds with `'content-type': 'text/html'`.

This is standards compliance behavior: http://en.wikipedia.org/wiki/ISO/IEC_8859-1#History

You can do the following to ignore server-set encodings:

``` python
r = requests.get('http://www.stanford.edu/dept/visitorinfo/activities/dining.html')
r.encoding = None
```

Then, the contents of `r.text` will be accurate, as the chardet detection will kick in.


If there is a BOM, wouldn't it be reasonable to override `text/html`?
Support for streaming POST data

This used to work in previous versions of requests (I think I was on 0.8 before).  I had something like this in my code:

  headers = {'Content-Length': str(cmd.getLength())}
  requests.post(..., data=cmd.genPayload(), headers=headers)

Where genPayload() was a generator that yielded the chunks of payload based on the command format.

In the latest builds of requests, I get a ValueError stating that it is "Unable to encode lists with elements that are not 2-tuples.".

I could set data="".join(cmd.genPayload()), but for large commands, this causes all the data to get buffered into memory.

Is there a way to support streaming POSTs with requests?


This is probably related to #635


Not yet, but planned for the near future :)


Why did this get closed, I just re-tested on 0.13.6, and I'm still getting:

`ValueError: Unable to encode lists with elements that are not 2-tuples.`


Streaming uploads are in the mid-term roadmap.


Got it.  Is there a way I can track that work / milestone?


Well, this is the plan:

http://kennethreitz.com/the-future-of-python-http.html


Armin and I need to get started :)
Auto-detect CA Cert path instead of requiring users to hack setup.py

Also fixed nearby comments to wrap at 79 characters as per PEP8


Closing.

I want everyone to be using certifi for a large number of reasons. The only reason I have the `get_os_ca_bundle_path` function is for people with heavily managed systems.

If they don't want to install it, they can specify `--no-deps`.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1528603) (merged 62a23bfb into b026def9).
Bug 637

Should solve #637.


Is this better than setting `'Content-Type': ''`?


@travisbot ?


"Is this better than setting 'Content-Type': ''?"

But, in #637, you told me that ignoring content-type as None is categorized as a Bug.

Settings as empty string should solve the dataXform problem.

Nevertheless, both of the proposed solution still lack proper multi-dict urlencoding handling.

If it is a design decision, then I think it should have some note on the docs regarding that.
If it is not, then it is another Bug.


Can you reopen? I was just asking to see if anyone had any opnions :)


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1527630) (merged fe2a0747 into b026def9).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1536918) (merged 176d26f4 into 62a23bfb).
Change log level to debug for connection pool operations.

Please?


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1504749) (merged ce71b722 into 355b9716).


This is an issue that should be raised with urllib3.


@lgastako See also: https://github.com/shazow/urllib3/pull/71

If you have anything to add to the conversation, I'd like to hear it. :)
Implement GSSAPI/Kerberos authentication for requests

Currently has two issues which may be related:
- doesn't work with redirects, we try to authenticate the server a second
  time with a fully completed kerberos context.
- 403 responses result in the wrong response object being returned, but
  this is also true for http digest authentication due to a bug in hook
  handling in general.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1503217) (merged c803e3af into 355b9716).


This looks awesome! Do you know of any way i can test this?


Unfortunately no, not without a kerberos kdc and a web server setup with gssapi/kerberos support.


Oh, nice. I just downloaded this, and it works just fine against the internal server I was using while writing http://python-notes.boredomandlaziness.org/en/latest/python_kerberos.html

For an automated test, you could look at doing something based on the python-kerberos unit tests: http://trac.calendarserver.org/browser/PyKerberos/trunk/test.py

Specifically, if you look at testGSSAPI, the server side of that could be pulled out into a handler in a temporary HTTP server that extracts the Authorization header from the request, calls GSSServerStep and GSSServerRespone, and then sets the WWW-Authenticate header on the response.

Mix and match to cover the following three cases with mutual_auth enabled and disabled:
- server sets the WWW-Authenticate response correctly
- server doesn't set the header at all
- server sets the header to an invalid value

One use case that this doesn't yet cover is that once a browser has seen the Negotiate header for a particular URL, _all_ future requests to that URL will have the header added (which can be damn annoying if you want to test dropping your connection!). A custom client like mine which _knows_ the server is expecting Kerberos also doesn't want to wait until the 401 - I want to force the transmission of the Kerberos ticket details immediately.

Perhaps an extra "always_auth" flag? Either that or two different authentication handlers:
HTTPKerberosAuth - always sets the Authorization header
HTTPNegotiateAuth - does the 401 dance before setting the Authorization header


Can't wait for this to be pulled, so that I can pull. :)

Thank you, @ncoghlan for the write-up.


Merging! Looking forward to further improvements :)


Huzzah!

Just a follow up on the test situation - I realised my idea wouldn't work, as both the client and server side of the GSS API code expects the Kerberos infrastructure to exist.

However, I realised that what _could_ be done for regression test purposes is to stub out the GSS API calls with an object that keeps track of the call sequence to ensure it doesn't change unexpectedly. That way, if the tests _did_ break, Kenneth would know that it needed to be rechecked against a real Kerberos setup to make sure it was still working.


I don't test OAuth at the moment either, and I don't want to officially maintain it, so let's just let twitter/github issues be the test suite in the meantime ;)


Thanks!
Allowing hooks to return responses that indicate errors

Since response objects for failures (4xx/5xx responses) evaluate to False
in a boolean context, any hook which returns such a failure response will
evaluate to False.

The way hooks were setup, any failure response resulting from a hook would
be ignored, and the initial response before it got processed by the hook
would be substituted in its place. This commit changes that logic to test
for None so that hooks that return failures can do so.

This would close out issue 644 (https://github.com/kennethreitz/requests/issues/644) which I opened earlier. It has some discussion about my findings, so you may want to look there for reference.

I've run the tests before and after this and the only tests that break as a result of this (as far as I can tell since I'm running python 2.6) are those in test_DIGESTAUTH_WRONG_HTTP_401_GET. That's because the bug that this fixes was hiding a bug in httpbin which is returning a 403 instead of a 401 on a failed digest authentication (it gets http basic right). I've submitted a pull request for httpbin that should address that (https://github.com/kennethreitz/httpbin/pull/48).


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1502992) (merged 662c3eda into 355b9716).


Yay for travisbot. As expected, it looks like the only test(s) that are failing are the tests associated with test_DIGESTAUTH_WRONG_HTTP_401_GET.


Excellent find! :cake:


http://travis-ci.org/#!/kennethreitz/requests/builds/1533375 :)
Using unittest2 where available for running tests

We're using some features that weren't available in unittest in python
2.6, but they are available in unittest2 which can be pip installed.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1502951) (merged 54df76f0 into 355b9716).


Instead, I think we should just use the features available in Python 2.6, unless there's a good reason not to.


That could be a significant amount of effort.


Can you give an example?


Unfortunately I run 2.7 at home and can't test at the moment to see what specifically is incompatible with 2.6. I'll get back to you when I have some time at work to test on 2.6.

## 

Michael Komitee

On Sunday, June 10, 2012 at 1:30 AM, Kenneth Reitz wrote:

> Can you give an example?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/645#issuecomment-6226074
Is the redirect history consistent ?

I have written a small example to fetch http://www.google.com ( from France that is why there is a redirect )
So it seems the history is not really consistent or more exactly I would have expected the first redirect response object to have a value for the request.

Any thought? Is it bug ? In this case i can have a look to fix it or the expected behaviour?

```
urls = ["http://www.google.com"]

for url in urls:
    print 'Fetching %s' % url
    r = requests.get(url, prefetch=True)
    history.append(r)

for r in history:
    print r.request.url
    print r.url
    for redirected in r.history:
        print redirected.request
        print redirected.url
```

Output
http://www.google.com
http://www.google.fr/
None  !!!!!
http://www.google.com/


you should check `r.history`


This is what i use. If you look carefully to the line "for redirected in r.history:".
fix for #630 - unquote_unreserved raising ValueError on invalid URI-esca...

...pe sequences

Especially relevent to security testing, an HTTP client will be asked to send requests containing invalid uri-escape sequences. This raises a ValueError if an attempt is made to parse a string argument as a signed integer with radix 16. The method unquote_unreserved attempts to unquote any uri-escaped sequences in the unreserved character set. Because a uri-escaped character is hex-encoded, we can simply add a check to ensure the sequence is alphanumeric prior to parsing.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1456398) (merged d60970ea into de1637c3).


Merge conflict, so added manually.

---

Thanks!
safe mode doesn't work when not returning response

``` python
from requests import get

if __name__ == '__main__':

    invalid_url = 'http://_/'
    req = get(invalid_url,
              config=dict(safe_mode=True),
              return_response=False)
    res = req.send() # throws ConnectionError
```

The reason is catch_exceptions_if_in_safe_mode() decorates request() which calls Session.request() which does not call Request.send() when send_request is False which is the case with async requests. I guess we should decorate Request.send() not request()

See issue #583


We will also have to change the catch_exceptions_if_in_safe_mode decorator a bit, so that it can decorate a method rather than a function, but that shouldn't be difficult.


+1 for this @piotr-dobrogost requests fork fixes exceptions when using grequest


an obsolete request since safe_mode is removed in 1.0?
no_proxy support

Adds support for no_proxy environment variable


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1449524) (merged b8561cda into de1637c3).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1454295) (merged cb8f9756 into de1637c3).


This is great. Thanks!
Dict data is not being properly urlencoded

I was trying to use FB Graph API and notice a little problem.

Consider the following example:

``` python
destination = 'https://graph.facebook.com/me/feed/'
info = {'access_token': '1234567890', 'message': 'TEST', 'privacy': {'friends': 'NO_FRIENDS', 'value': 'CUSTOM'}}
requests.post(destination, data=info)
```

The above example will be rejected by FB Api (error400).

At first, I thought it was a content-type problem, because of the default application/x-www-form-urlencoded
that requests set as content-type when post data is not a string.

So, I just change the above example to force content-type header to be None: 

``` python
...
headers = {'content-type':  None}
requests.post(destination, data=info, headers=headers)
```

But, since I'm not making info a string, requests keep trying to send a content-type=application/x-www-form-urlencoded

Recently, I've tried with success to mix urllib with requests and discover that
although I can't force a content-type header to be None (is this another issue?),
the problem with FB rejecting my Post is related with a wrong urlencoding of post data.

This is the example version which actually is working (and I'm using in my code):

```
import urllib
info = urllib.urlencode(info) 

requests.post(destination, data=info, headers=headers)  # profit
```

urllib.urlencode transform info into 
'access_token=1234567890&message=TEST&privacy=%7B%27friends%27%3A+%27NO_FRIENDS%27%2C+%27value%27%3A+%27CUSTOM%27%7D'

and this string is correctly accepted as a post request by FB.


If you set the value to `None`, we shouldn't be assigning it at all. This is a bug.

Thanks for finding it :)


So...
Are you saying that the `headers = {'content-type':  None}` being ignored is the proper Bug ?

Or .. are you referering to both "content-type" being ignored AND data being not properly urlencoded
 as a whole bug ?

Thanks


The first :)


I see.
I should investigate the causes on the next weekend 
and if anything useful has been found, propose some commits to your evaluation.


I just sent a pull request fixing the `content-type=None` issue described above.

But, requests still have some trouble trying to urlencode multi-dict data.

For example: 

``` python
data={'another_dict': {'a': 10, 'b':100}}
```

I'm having a little hard time trying to figure out if this is actually a bug
or some design decision (let's say that: If you want a multi-dict data, then you should provide an already formatted
urlencoded string rather than use the built-in urlencode feature, which should be only used to  encoded form data).

What are your thoughts on this ?


@rafaelmachado since we're handling headers as a case-insensitive dict, shouldn't you do `if v is not None and k.lower() != 'content-type'` instead?


Very true. You are probably trying to remember me about RFC2616 
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html).

Thanks.
Unable to POST data and files

I found a logic error while doing something experimental:

If you call `requests.post(data="urlencoded string")`, everything works.

If you call `requests.post(data="urlencoded string", files=<dict>)`, requests will raise a `ValueError` in `_encode_files`:

https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L349

By that time `str` in the local scope is actually `unicode`, so the `isinstance` check fails. It seems like this should either be `isinstance(self.data, basestring)` or simply raise something like `TypeError` to indicate that attempting to send both a pre-encoded string and files is pointless and should not be done.


+1 for `(TypeError, RequestsException)`


`data=<dict>, files=<dict>` should work fine. 

I don't think injecting content into a fully-provided request body is a good idea. Feel free to re-open if I misunderstand.
poor man's patch for issue #630

```
this try catch is a poor man's patch for issue #630
https://github.com/kennethreitz/requests/issues/630
```


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1418645) (merged 7125fe55 into de1637c3).


Thanks!


Hey Kenneth, I would recommend against wrapping the whole method within a try/except ValueError block. I think my pull request #641 should be sufficient in addressing @tzuryby's issue. If there is another edge case that would raise an exception here, it would be valuable for us to understand why it did so.  Catching the exception and continuing would prevent us or the developer from even knowing there was an issue with the URI.


> I would recommend against wrapping the whole method within a try/except ValueError block

Agree.


that's a far better fix than my catch-all approach, no doubt.
that's why I called it "poor man's patch.." ;-)
Issue #505: Changed the store_cookie configuration to be part of the conig dict, instead of being a new argument

Hi,

Reforked the repository to do away with the git mess I had created. Hope this works this time.

Thanks,
Arup


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1416011) (merged b2867336 into de1637c3).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1416113) (merged 4a309796 into de1637c3).


Hi @kennethreitz,

Any update on this pull request? I understand if you are busy with other things.
Jus trying to "bug the maintainer" :) 

Also let me know if this requires any other change.

Thanks,
Arup  


@amalakar :cake: :)


This is perfect. Exciting :)


Yay! Thanks @kennethreitz for the encouraging words. We can close Issue #505 now. I need to find the next bug to fix now. Feel free to assign/recommend something :)
invalid literal for int() error for complex URIs

I am using requests to perform security tests against my site (in this case, a simple sql-injection vector) and getting 
error (dumped below)  which points to `unquote_unreserved` function.

see below

snippets of my code:

```
#   tuple structure: 
#   (path, data, headers, cookies, files)
(r"""/default.asp?id=1+AND+1=(SELECT+TOP+1+1+FROM+information_schema.tables+WHERE+TABLE_SCHEMA="blind_sqli"+and+table_name+NOT+IN+(+SELECT+TOP+1+table_name+FROM+information_schema.tables)+and+table_name+LIKE+'[a-z]%'+)""", {}, {}, {}, {})

...

requests.get(options.host + path, data=data, headers=headers, cookies=cookies, files=files)...


```

The exception dumps

```2012-05-23 15:10:15,330 - ERROR - invalid literal for int() with base 16: "'+"
Traceback (most recent call last):
  File "sqli.py", line 460, in run_case
    return fetch[method](options.host + path, data=data, headers=headers, cookies=cookies, files=files).status_code
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 52, in get
    return request('get', url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 40, in request
    return s.request(method=method, url=url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 229, in request
    r.send(prefetch=prefetch)
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 468, in send
    url = self.full_url
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 411, in full_url
    url = requote_uri(url)
  File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 448, in requote_uri
    return quote(unquote_unreserved(uri), safe="!#$%&'()*+,/:;=?@[]~")
  File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 429, in unquote_unreserved
    c = chr(int(h, 16))
ValueError: invalid literal for int() with base 16: "'+"

```
```


btw, this is my quick hack

```

def unquote_unreserved(uri):
    try:
        """Un-escape any percent-escape sequences in a URI that are unreserved
        characters.
        This leaves all reserved, illegal and non-ASCII bytes encoded.
        """
        parts = uri.split('%')
        for i in range(1, len(parts)):
            h = parts[i][0:2]
            if len(h) == 2:
                c = chr(int(h, 16))
                if c in UNRESERVED_SET:
                    parts[i] = c + parts[i][2:]
                else:
                    parts[i] = '%' + parts[i]
            else:
                parts[i] = '%' + parts[i]
        return ''.join(parts)
    except:
        return uri
```


@tzuryby so make a pullrequest !
Retry does not occur during timeout

Why does this not retry 3 times?

```
import time
import logging
import requests
from requests.exceptions import Timeout

logging.basicConfig(**{
    'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    'level': logging.DEBUG
    })
logger = logging.getLogger()

TIMEOUT = 2
RETRY = 3
start = time.time()
try:
    r = requests.get(
        'http://httpbin.org/delay/5',
        timeout=TIMEOUT,
        config={'max_retries': RETRY}
    )
except Timeout:
    print 'Timed out, sad trombone.'
stop = time.time()
print 'took: {}s'.format(stop - start)
```

$ python test.py 
2012-05-22 16:07:34,023 - requests.packages.urllib3.connectionpool - INFO - Starting new HTTP connection (1): httpbin.org
Timed out, sad trombone.
took: 2.0639898777s


Will the timeout setting cause requests to time out when it can't **connect** to the server within the specified time, or does it time out when it can't finish reading the response within the specified time. Based on my example it seems that it is the former, not the latter.

If this is the case, then would it be imprudent to set the timeout to something rather low and attempt to retry if there is a Timeout raised? I am trying to make my code as robust as possible, and on occasion the 3rd party API I'm accessing via HTTPS raises a Timeout or SSLError exception (something like read timed out). I want to fail as fast as possible if it's going to timeout, then retry a few times before giving up.


timeout is for connects.
Improvements to RequestsCookieJar

- Docstrings for all methods in RequestsCookieJar
- Defined keys(), values(), and items() in order to let user convert RequestsCookieJar to a plain Pyton dict
- If there are multiple cookies with the same information, we now throw CookieConflictError
- Defined a couple of small utility functions

Discussion here: #578
I decided to only throw CookieConflictError if there are actually are conflicting cookies (it is fine to have multiple domains in jar), in order to not break any existing client code.


Beautiful! We'll see what @travisbot says :)


This pull request [fails](http://travis-ci.org/kennethreitz/requests/builds/1396842) (merged 96cd8e9c into 2d974662).


Looks like it breaks on Python 3 :(


I left a print statement in there from when I was debugging. Nice catch, travisbot.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1396868) (merged 0eb0b894 into 2d974662).


:cake:

Thanks for this. Excellent improvements all around :)
SSLError error raise for https urls (SSL23_Get_Server_Hello:unknown protocol)

Hi, 
I'm trying to acces github api v3 through requests, but I get the attached traceback when trying to connect to any https urls.

As I'm behind a proxy, I tried setting the proxy as shown in another issue, but proxy respond to me that it can't get the URL.
I also tried `verify=False`, which raise Timeout
I'm surely dooing something wrong, so if I can get some clues of what... 

Thanks.

( OS X 10.7.3 , http(s)_proxy exported in bash)

Without setting proxies

``` python
In [43]: requests.__version__
Out[43]: '0.12.1'

In [32]: c = requests.get('https://api.github.com/')
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
/Users/matthiasbussonnier/ipython/<ipython-input-32-f476e1ab2b17> in <module>()
----> 1 c = requests.get('https://api.github.com/')

/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests-0.12.1-py2.7.egg/requests/api.pyc in get(url, **kwargs)
     51     kwargs.setdefault('allow_redirects', True)
---> 52     return request('get', url, **kwargs)


/usr/.../requests/api.pyc in request(method, url, **kwargs)
     39     s = kwargs.pop('session') if 'session' in kwargs else sessions.session()
---> 40     return s.request(method=method, url=url, **kwargs)

/usr/.../requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, return_response, config, prefetch, verify, cert)
    228         # Send the HTTP Request.
--> 229         r.send(prefetch=prefetch)

/usr/.../requests/models.pyc in send(self, anyway, prefetch)
    607                 except (_SSLError, _HTTPError) as e:
    608                     if self.verify and isinstance(e, _SSLError):
--> 609                         raise SSLError(e)

SSLError: [Errno 1] _ssl.c:503: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
```

with setting proxy

``` python
In [33]: PROXY_LIST = {                                                    
      'https':'www-cache:3128'
}

In [36]: c = requests.get('https://api.github.com/pulls',proxies=PROXY_LIST)

In [37]: print c.content # translated by me from french
[...]
Error :
Don't support this combinaison of methods and protocol for following URL
https://api.github.com/pulls

Squid does not allow all methods for all URLs, for example, you can't use POST in a GOPHER request.
[...]
```


What version of openssl do you have installed?

`openssl version` in a terminal


OpenSSL 0.9.8r 8 Feb 2011.

I thought of upgrading (to 1.0.1c) with homewbrew but  it says not to :

```
This formula is keg-only.
Mac OS X already provides this program and installing another version in
parallel can cause all kinds of trouble.
```

Maybe there is a way to tell requests to use a specific version ...

Don't know it it helps, but, it works with curl...

Thanks.


This might be related with #606


Closing for #606
Documentation Translation Guide


Yes we need it because I might translate the documentation to arabic sometime


@toutouastro that would be fantastic!


Japanese is underway: http://ja.python-requests.org


The summer break is coming and I'll hack on this project a lot this summer !


How popular is `requests` among the Hispanic community?


Awesome! I cannot wait :)


Any advise on how I should start?


I'd be willing to do a Danish translation.


Thanks for the all the help, guys! 

I created a bunch of repos here: https://github.com/requests

I've created a new 'Translations Team', and added all of you to it. You should be able to push to any of the translation repositories. :)

:sparkles: :cake: :sparkles:


Am I doing it right?
What do I need to do to have the Spanish version published like the Japanese one? (http://ja.python-requests.org)

Thanks


@kennethreitz any chance for a Portuguese repository? :)


@duailibe are you volunteering? :)


@kennethreitz yes! :)


I'm on it!


@duailibe thanks so much! Just push here :) https://github.com/requests/requests-docs-pt


That's funny. A lot of translating going on, but the translating guide never got written. 
I'm basically translating anything under docs/ 


@kennethreitz awesome! Thanks! :cake: 


@serpulga I'd like to contribute to the Spanish translation too. I see you're doing it now, but I think some things have to be changed, and I'd also like to help you finish the entire translation if you haven't. What should be the procedure?


Hi @omederos. I think regular git-flow / pull requests would be the way to go.
I think I pushed the one document that was left, and I honestly don't know
what's next. Let's see if we can get some pointers from @kennethreitz  for the
other translators.

Thanks,

Sergio


I’d love to help out with Polish.  Requests is awesome and it needs more docs translations.


@Kwpolska all set! Let me know when you have an update so I can make it official :)

https://github.com/requests/requests-docs-pl

Thanks so much :)


Can I volunteer to translate the docs to Indonesian, please?


Can I ask what does it need to put the spanish documentation online, @kennethreitz ? 
(Besides some update, it has 9 months old, as I see here: https://github.com/requests/requests-docs-es)


I can translate to Turkish. How do i start?


@deusExCore start a fork and start translating the documentation in place. When we can grab his attention, we'll have @kennethreitz add you to the requests organization and help you move your fork to github.com/requests/requests-docs-tr and then he, or I will have to set up DNS to resolve that for you


Hi guys, here you have a candidate for translating the docs to Italian!
@kennethreitz @sigmavirus24 if it's a good thing, I can start working on that on a fork


@csparpa go for it!


yep :)


@kennethreitz would you setup a specific repo under https://github.com/requests for the IT translation? I'll wait for that if you're going to create it


@csparpa please start translating and as soon as it's mostly done is when we usually import your fork into the @requests org
fetching small part of data instead of whole file

Correct me if i`m wrong but it`s not able to fetch only small part of data instead whole file.

for example if i want to download large file and save to hdd, whole file have to be first buffered in ram:

<pre>open.write('large_file', 'wb').requests.get('http://large_file').content)</pre>


it should be like that:

<pre>file = open('large_file', 'wb')
connection = requests.get('http://large_file')
while True:
     buffer = connection.fetch(1024)   # fetch only 1024 B
     if not buffer:
          break
     file.write(buffer)
file.close()
</pre>


You should take a look at `response.iter_content`
Changed the store_cookie configuration to be part of the config dict, instead of being a new argument

Hi kennethreitz,

I have changed the code to make the configuration part of the config object instead of being a parameter.

This is the actual commit: 

https://github.com/amalakar/requests/commit/ae3b13920875c3068a19f04c3bc8f7d4cbdfd76f

Git issue: After realizing you had issue merging yesterday I did some 'rebase' and this pull request now has 17 other commits :( I am not sure if this can be merged. If not I will try something else. I come from svn world and looks like I need to learn git. My way of learning via trial and error is not working well here.

Thanks,
Arup


Hmm, looks like you aren't based off the current develop branch.


Let me close this and send another pull request.
Getting HTTP 400 errors when trying to POST data and files

Whenever I try to POST a form with data and files, I get HTTP 400 (Bad request) errors. I can post same data and to same URL w/o problems through a web browser.


is the web browser doing multipart?


I believe so. Here's the gist https://gist.github.com/2720779


I think I see the potential issue: if you look at the browser headers above, you see that the Content-Type is set to multipart/form-data when I do the request through requests, the headers show Content-Type 'application/x-www-form-urlencoded' which might be causing the server to refuse that content type? 


I had a similar problem trying to post json strings.  The problem went away when I set Content-Type to 'application/json'.

It seems that this problem happens because post defaults to x-www-form-urlencoded even though the input param is a string (I checked using httplib2 with no headers, and the same server was able to accept).
Do not send cookies marked with secure flag over insecure HTTP connection

By default, the cookielib.CookieJar used by requests does not honor standard accept/reject rules for Netscape and RFC295 cookies. This behavior could result in the inadvertent exposure of sensitive cookies that have been set with the 'secure' flag (i.e., they are only to be sent to HTTPS resources) when requesting non-HTTPS connection.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1358411) (merged 3f99dfd2 into 2d974662).


Woops, my apologies. The version of requests that I had tested against was not using cookielib.CookieJar.


For the record, this is tested by `test_secure_cookies` here: https://github.com/kennethreitz/requests/blob/develop/tests/test_cookies.py
Rewrite quickstart docs.

Issue #503 raised a few problems with the lack of clarity in the quickstart documentation. This pull request rewrites that documentation, to cover the issues raised in #503, and for general clarity.

If there are other docs rewrites you'd like, or changes to this one, I'd be happy to update this pull request with further commits.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1354039) (merged 45506d1f into 2d974662).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1358709) (merged 36fd8da1 into 2d974662).


This pull request now also addresses issue #504.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1376845) (merged 2b8e01bf into 2d974662).


@kennethreitz: How do you feel about that last commit as a first draft of a solution to #482?


Thanks for this! I'm :-1: in a few things in here, but I can tweak later :)
Issue #505: Allow disabling of cookie persistence/sending

I tried to add the feature as requested in issue #505. When store_cookies is set to False, all the cookies sent by the server would be just ignored. The user can still provide cookies using the cookie param though and they would be passed to the server as well. Just that the ones sent by the server are ignored.

Thanks,
Arup


Hmm, was this pull request against latest? Looks like GitHub thinks that it can't automatically be merged.


I am new to github and I am lost what is missing here :( Shall I repeat the process?


Hmm, I'm not sure, honestly. Everything looks good to me.


Excellent, thanks for the change :)


Actually, I should have looked a bit closer at this. cookies should be added to config, not added as a new argument.


Well it was my mistake I now realize that you mentioned as "The second one (with config) is definitely best". Let me fix the to use the config object. Then I will send a new pull request. Does it work?


Perfect!
Expose case-insensitive key access when deleting a key from CaseInsensitveDict

Previously, when attempting to call del on a dictionary key that did not exist, a KeyError was raised. This commit changes the default behavior to be case-insensitive.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1348101) (merged 4b7c5dab into c477f6eb).


Awesome, thanks!
draft pull request for eager release of sockets

Please do not merge this! This is intended as a "design review" :-) It needs more comments and tests before it is production ready.

This implements explicit disposal of sockets in urllib3 and Requests that does not rely on instantaneous garbage collection via reference counts (e.g., for PyPy). It should fix #520 without removing the `Response.request` member, or otherwise breaking the API (or the reference cycle, depending on whether you're a glass-half-full or glass-half-empty person).

It's split into three commits: one is a test case, one adds explicit disposal, one defaults `prefetch=True` and changes the tests to match.

@shazow maybe you could take a look at the urllib3 changes?

Thanks everyone.


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1341277) (merged e490a2f9 into 355003b4).


This pull request [passes](http://travis-ci.org/kennethreitz/requests/builds/1343119) (merged b4556066 into 355003b4).


Could you make a separate pull requests just for urllib3? I'll take a closer look at it there.


I'm closing this in favor of discussion at shazow/urllib3#87.