Raibek/requests-github-issues huggingface dataset:
Display full URL when logging.

Logging should display the query string as well, shouldn't it?

Test script:

``` python
import requests
import sys

my_config = {'verbose': sys.stderr}
my_params = {'q': 'test'}
r = requests.get('http://www.google.de/search', params=my_params, config=my_config)
```

Before:
`2011-11-20T10:13:37.502444   GET   http://www.google.de/search`
After:
`2011-11-20T10:14:02.652295   GET   http://www.google.de/search?q=test`


I agree — thanks!
some additions to .gitignore

added Eclipse project files and .egg-info dir
entries sorted alphabetically (case-insensitive)


I'll add the `.egg-info`, but not the ecplise files. Those belong in your [global gitignore](http://help.github.com/ignore-files/) :)


So what's `.idea` entry doing there?


It should absolutely be removed :)


Global gitignore wasn't known when I started this :)
Custom authentication is broken / incorrect docs

# Teh broken

Consider the following code, mostly lifted from the docs:

``` python
def pizza_auth(r, username):
    """Attaches HTTP Pizza Authentication to the given Request object."""
    r.headers['X-Pizza'] = username
    return r

import requests
r = requests.get('http://www.google.com', auth=(pizza_auth, 'foo'))
print r.request.headers['X-Pizza']
```

It appears that basic auth is hardwired for cases where two params are passed ([source](https://github.com/kennethreitz/requests/blob/master/requests/auth.py#L133-134)).

Either I'm not getting something, or auth doesn't work as documented / the way I expect.
# A proposal

Why not make all auth handlers callable objects? For example:

``` python
class AuthBase(object):
    """Base class that all auth implementations derive from"""

    def __call__(self, r):
        raise NotImplementedError('Auth hooks must be callable.')


class HTTPBasic(AuthBase):
    def __init__(self, username, password):
        self.username = str(username)
        self.password = str(password)

    def __call__(self, r):
        auth_s = b64encode('%s:%s' % (self.username, self.password))
        r.headers['Authorization'] = ('Basic %s' % auth_s)
        return r


# Inline usage
r = requests.get('http://foo.com', auth=HTTPBasic('idan', 'secret'))

# Also in recyclable form!
idan_auth = HTTPBasic('idan', 'secret')
r1 = requests.get('http://foo.com', auth=idan_auth)
r2 = requests.get('http://bar.com', auth=idan_auth)
```

Such an approach has a few benefits:
- Doesn't make the trivial use-case's syntax any worse (if anything, it seems more explicit/clearer to me)
- Makes reuse much simpler if you're not starting a session.
- Facilitates implementation of more complex forms of auth, like OAuth

For the latter case, I could do something like:

``` python
idan_oauth = OAuth(consumer_key, consumer_secret, access_token, access_token_secret)
requests.get(twitter_api_url, auth=idan_oauth)

# or even
client = session(auth=idan_oauth)
r1 = client.get(an_endpoint)
r2 = client.get(another_endpoint)
```

… which would keep the OAuth implementation details neatly encapsulated inside the OAuth class.

This would also make adding OAuth (#65) relatively easier.

Thoughts?


Here's an example implementation of header-based OAuth: https://gist.github.com/1373019

I would probably break it into an OAuthBase class with the common methods, and then subclass for every OAuth parameter method (header, querystring, body).

All in all, it's only ~100 loc.


Sorry for the delay on this — carrying conversation over to #276


Closing for #276
Correct handling of URLs with quoted slashes.

The Request class currently does a urllib.quote(urllib.unquote(path)) cycle to ensure consistent quoting on the input URL.  Unfortunately this breaks paths that contain quoted slashes (i.e. "%2F") since they get unquoted by urllib.unquote but are re-quoted by urllib.quote.

This patch adds a requote_path() function that does the same thing but is careful not to destroy quoted slashes.


still broken. try to run

``` requests.get('http://www.cocktail.uk.com/db/random.asp')
```

should have 404


This appears unrelated to my change, as it also gives a 404 using the latest release build.  In any case, I've pushed a change that makes it work for me.


Oh man, thank you ever so much.
Now it works for me ;)


Excellent — thanks!
allow_redirects=True by default also for post-requests

I think it makes sense to also have allow_redirects=True as default for POST-requests.

I have seen Ruby on Rails application in the wild which often directly redirect after a POST.


Absolutely not. Only 303 responses should be treated this way.

Please see http://tools.ietf.org/html/rfc1945. 


If we agree to implement change requested in issue [#269](https://github.com/kennethreitz/requests/issues/269) we should also implement automatic redirects for 301 and 302 redirects after POST request as per my [comment](https://github.com/kennethreitz/requests/pull/269#issuecomment-2883086)
Proxy dose not work.

My  test Code:

``` Python
import requests

r = requests.get('http://www.google.com', proxies={'http': '127.0.0.1:888'})
print r.content
```

proxy, {'http': '127.0.0.1:888'} does not exist,so there should not return the correct results.
Looks like not using a proxy.
version: 0.8


In addition to that, proxies are not used in redirects.


Fixed!


Need to find a way to test proxies btw.


I use Charles http://www.charlesproxy.com/ .


I meant in the testsuite.


v0.8.1 was just released, which fixes this!
Timeout ignored when following redirects

To reproduce, add a print to requests.models.Request.send():

```
def send(self, anyway=False, prefetch=False):
    """...."""
    print "send with timeout %s" % self.timeout
```

Then:

```
>>> requests.head('http://httpbin.org/redirect/3', timeout=10)
send with timeout 10
send with timeout None
send with timeout None
send with timeout None
<Response [200]>
```

I expected a timeout of 10 everywhere, not None.


(Sorry for the closed/reopened -- me typing in the wrong window resulted in accidentally closing the ticket)
Timeout exception not being thrown - NoneType instead

On OS X, I'm not getting the nice Timeout exception. 

``` python
>>> requests.get('http://github.com', timeout=0.001)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/api.py", line 78, in get
    return request('get', url, **kwargs)
  File "requests/api.py", line 65, in request
    prefetch=prefetch
  File "requests/sessions.py", line 187, in request
    r.send(prefetch=prefetch)
  File "/Library/WebServer/Documents/lab/python/requests/requests/models.py", line 418, in send
    timeout=self.timeout,
  File "requests/packages/urllib3/connectionpool.py", line 342, in urlopen
    self.timeout)
TypeError: float argument required, not NoneType
>>> requests.get('http://github.com', timeout=1.0)
<Response [200]>
```


Minor bug in urllib3 — thanks!


v0.8.1 was just released, which fixes this!
Make it possible to choose multipart encoding instead of form encoding

Currently multipart (multipart/form-data) encoding is used only for files. It should be possible to use it instead of form encoding (application/x-www-form-urlencoded) in every place the latter one can be used.


Agreed! I've been wanting this for a while. Coming up with an elegant api is tricky though.


As much as I love elegant code I'm afraid that after reaching some level of complexity one has to leave elegance to others :( Could we just add yet another param (called transfer_encoding) to request method?


Requests is 100% about API elegance. The API comes before all else. 

I have a few ideas about what it would look like, but I'm letting it stew for a little while.

In the meantime, you can multi-part encode yourself and upload the data as a bytestring.


> I have a few ideas about what it would look like  

Please share them with us so that we could evaluate them.

> In the meantime, you can multi-part encode yourself and upload the data as a bytestring.  

That's what I'm doing now using Poster and urllib2. I hoped for shorter code when switching to Requests :)


Add a fake file.


Deferring for now. Will reopen soon.
Support forum

It would be nice to have some support forum for users.


You're on it :)

## 

Kenneth Reitz

On Nov 13, 2011, at 5:20 PM, Piotr Dobrogostreply@reply.github.com wrote:

> It would be nice to have some support forum for users.
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/issues/257


Am I really? :)
I proposed this because I feel the best approach is to have both a forum and a tracker. Asking questions on tracker seems an awkward idea. Setting up google group is very easy.


I can setup some phpbb or vbulletin if you want. just let me know


We actually already have a mailing list, but it's not in the docs. I'll add a link in the docs.

Just send an email to `requests@librelist.com` :)


I think a google group would have much more friction.


Give the existing mailing list a try, please. Librelist is quite wonderful.
GET request sends full url to server - should be path only

The command `requests.get("http://www.raumzeit-podcast.de/feed/")` results in 

```
GET http://www.raumzeit-podcast.de/feed/ HTTP/1.1
Host: www.raumzeit-podcast.de
Accept-Encoding: identity, deflate, compress, gzip
Accept: */*
User-Agent: python-requests/0.8.0

HTTP/1.1 301 Moved Permanently
Location: http://www.raumzeit-podcast.dehttp/www.raumzeit-podcast.de/feed/
Content-Encoding: gzip
Content-Length: 20
Content-Type: text/html
```

Requests should send `GET /feed/ HTTP/1.1`

Version: 0.8


Confirmed this is an issue. Same issue arises on my rss feed as well

``` python
>>> r2 = requests.get("http://www.vertstudios.com/blog/feed")
>>> r2.url
'http://www.vertstudios.comhttp/www.vertstudios.com/blog/feed'
```


I also confirm. It's easy to reproduce:

-) in shell A run 

```
python -m SimpleHTTPServer
```

-) in shell B run 

```
curl http://localhost/this/is/a/test
```

in shell A you see "GET /this/is/a/test HTTP/1.1" in the access log, now

-) in shell B run

```
python -c 'from requests import get; get( "http://localhost:8000/this/is/a/test" )'
```

in shell A this time you have "GET http://localhost:8000/this/is/a/test HTTP/1.1" in the access log.


Apparently the culprit is utlib3, you get the same wrong request using directly

```
python -c 'from urllib3 import PoolManager; http = PoolManager(); http.request( "GET", "http://localhost:8000/this/is/an/example" )'
```

I'm not an urlib3 expert, but I suspect that `_make_request` in `urllib3/connectionpool.py` passes the full URL (scheme and host included) instead of the path (as shown in examples http://docs.python.org/library/httplib.html#examples). I'm filing the issue on urllib3 repo to see what happens…


Did it here https://github.com/shazow/urllib3/issues/8


Interesting. Nice find! Looks like some poor handing on the server, but something we should definitely fix.


Closing for #265


v0.8.1 was just released, which fixes this!


Really nice!
Ambiguous configuration settings

Couple of examples can be seen [here](https://github.com/kennethreitz/requests/blob/develop/requests/sessions.py#L66-67). `timeout` and `keep_alive` are both parameters by themselves but members of the config dict. I'm not sure if this was intended, but this allows the user to define these parameters ambiguously. I think either the keys in the config dict or the arguments should be removed. 


Don't worry, `keep_alive` won't be a parameter.  Still debating `timeout` though.
Max timeout won't trigger

Haven't looked at the code to see why yet, figured I'd post this here first.

``` python
>>> myconfig = {"timeout": 1.0}
>>> requests.get("www.google.com:81", config=myconfig)
```

The timeout never comes. 


So I misread the doc, timeout is actually passed as an argument to requests. This seems a bit weird, since timeout is both a possible key in the config= dictionary but also a default value in requests.defaults.

Regardless, the timeout still is not working. 

``` python
>>> r = requests.get("http://google.com:81", timeout=1.0)
```

This just hangs forever. 


After checking through the code, this may be a problem with urllib3. 

The following just hangs:

``` python
>>> import requests.packages.urllib3 as urllib3
>>> test = urllib3.HTTPConnectionPool("www.google.com:81")
>>> r = test.urlopen("GET", "/", timeout=1.0)

```

However, I'm not sure if timeout is ever passed to the urllib3 functions at all anyway (see [models.py](https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L398-408) and note the lack of `timeout` argument)


Yeah, timeout isn't implemented in the `develop` branch quite yet.


Google is a special case. It works properly on every other server I try. Diverting for now.


v0.8.1 was just released, which fixes this!


(not google)
Correct verbose logging documentation

Includes [Verbose logging example does not work](https://github.com/dasevilla/requests/commit/2d8440ea98301131edd537c4f2010b749f005e3b#commitcomment-712251), from @dasevilla and should close [issue #250](https://github.com/kennethreitz/requests/pull/250)


Thanks! However, I would like to discourage (by not documenting) the ability to override the 'defaults'. Global state = bad :)


Alrighty, closing the request. 


Sorry :)
Verbose logging example does not work


I incorporated your example into pull request [#251](https://github.com/kennethreitz/requests/pull/251).


Sorry, the docs are a little out of date :)  I would like to discourage (by not documenting) the ability to override the 'defaults'. Global state = bad :)


I understand this issue is closed but how can I enable verbose logging with requests v1.1.0? With earlier versions (v0.14.2), what worked well for me was: requests.session(config={'verbose': sys.stderr}). However, this no longer works. Apologies in advance if I did not read the docs enough.


All logging performed by requests was removed in the refactor. You can, however, turn on logging for urllib3. I think it is something like

``` python
from requests.packages import urllib3
urllib3.add_stderr_logger()
```

The default level for that logger is going to be DEBUG, so if you want INFO messages too you'll have to import logging as well and pass that function `logging.INFO`.


Thanks for the quick response. Will try out and look into urllib3 logging.
POST doesn't appear to be documented

It took me some time to figure out how requests.post worked, and the only mention appears to be in the API doc, and then not in great detail. Am I missing something?


http://docs.python-requests.org/en/latest/api/#requests.post


It definitely needs to be added to the quickstart.


As I said, it's in the API doc, but not in the main doc, where other much more complex topics are covered but not something as basic as requests.post.


Yes, please :-)


I'll put something together in the next couple of hours and see whatcha think.
malformed url when using digest with query-strings

When using digest auth and a URL containing a "?" the question mark is removed, it seems, from the url.

For example, 

r = requests.get('http://foo.com/bar.json', auth=('digest', 'blah', 'bloh'))

works, but:

r = requests.get('http://foo.com/bar.json?really=yes', auth=('digest', 'blah', 'bloh'))

Fails. Inspecting r.request.headers['uri'] shows:

'http://foo.com/bar.jsonreally=yes'

Of course, this could be just an output but, not the actual url that it's trying to hit. Unfortunately, I only have access to one side of this conversation, so I can't tell you what the library is actually sending/asking for. I'll try and find another server that I can test.


For URL parameters, use the parameters argument in your request.

``` python
>>> url = "http://httpbin.org/digest-auth/auth/user/passwd"
>>> auth = ("digest", "user", "passwd")
>>> params = {"really":"yes"}
>>> r = requests.get(url, auth=auth, params=params)
>>> r.ok
True
>>> dir(r.request)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_build_response', '_build_url', '_enc_data', '_enc_params', '_encode_params', '_get_opener', 'allow_redirects', 'auth', 'config', 'cookies', 'data', 'files', 'headers', 'hooks', 'method', 'params', 'proxies', 'redirect', 'response', 'send', 'sent', 'timeout', 'url']
>>> r.request.url
'http://httpbin.org/digest-auth/auth/user/passwd'
>>> r.request.params
[('really', 'yes')]
>>> r.request.auth
(<function http_digest at 0x1ea3e60>, ('user', 'passwd'))

```


Ok, I didn't see that in any docs. In fact, I don't see it now, other than the low-level API docs.

The spirit of requests is that it's supposed to be simple; a URL that works in a browser should work in requests, right? At the very least it shouldn't fail silently and corrupt the URL: I think it's still a bug.

In the mean time, I'll use the feature tip you pointed out. Thanks.


I agree that the documentation should cover that, and I'll try to get to it in the next couple of days.

As for your error, I was actually not able to replicate the disappearing '?'

``` python
>>> r = requests.get("http://httpbin.org/get?really=yes", auth=("digest", "blah", "bloh"))
>>> r.request.url
'http://httpbin.org/get?really=yes'
```

Try pulling from the develop branch and see if you still get the issue.
Timing HTTP Responses

I would like to be able to time HTTP responses, logging the url along with the time taken to a file

It seems like this should be reasonably easy with pre + post request hooks, I havent tried yet, this is mostly a feature request / asking if anyone else had already implemented this as I cant see a mailing list or anywhere to discuss these things.


Super simple :)

``` python
import time
import requests


time0 = time.time()
r = requests.get('http://github.com')
time1 = time.time()

print time1 - time0
```


heh yeh, I was looking more for the ability to inject the timing, as the requests code is buried inside various libraries and most of the time the timing wouldnt matter, but during load tests it would be useful

either way it isnt actually an issue, cheers


Ah, you could do it with hooks then, but that wouldn't be ideal.
AttributeError while trying to read `content` attr

It seems that if there's a problem with obtaining the body of a request, `requests` throws an error while trying to read it.

Code:

``` python
import requests

def main():
    r = requests.get("some-flaky-url")
    if not r.content:
        raise Exception('content was flaky')
    return r.content

main()
```

error:

```
Traceback (most recent call last):
  File "test.py", line 5, in main:
    if not r.content:
  File "eggs/requests-0.7.4-py2.6.egg/requests/models.py", line 499, in content
    self._content = self.raw.read()
AttributeError: 'NoneType' object has no attribute 'read'
```


It would be better to check the `ok` attribute on the response:

``` python
r = requests.get("some-flaky-url")
if not r.ok:
    raise Exception('content was flaky')
return r.content
```


Isn't it more pythonic to check what you're actually returning/working with (eg, the `content`)? One would expect that, in the case of something like a 304, `r.ok` would be `True` and `r.content` would be `None`; my example would return the required exception, whereas yours may cause failures further along.

I guess I could modify my example to check `if not r.ok and not r.content`, but I believe that would still raise the `AttributeError` in the same circumstance?


`r.content` should definitely be `None` in this situation.
Fixes an issue with auth and redirects

The `auth.dispatch` method creates a modified `auth` tuple. When the tuple is passed to a new request (as a result of a redirect) then the new `auth` is incorrect and authorization will fail. This one-liner simply resets the `auth` tuple to the original tuple.


After sleeping on it, I don't think this solution is the best one. I think a better solution is to save the original `auth` tuple (`_auth`?) and then pass that to any new requests.


Excellent, thanks!
Wrong default types for params and data

`params` and `data` are marked as dictionaries in the constructor prototype. However `_encode_params` returns lists or None sometimes.

There is a big inconsistency in `_encode_params` that could affect hooks. It's affecting me in request-oauth-hook. Thus I have to do these type of checkings:
https://github.com/maraujop/requests-oauth-hook/blob/dev/oauth_hook/hook.py#L184

So if params is a dictionary, I understand it should be an empty dictionary or a filled dictionary, but not a list or None. The same happens to data.

I haven't been able to run `requests_tests` sorry, they failed before I did a change to the code. This will likely affect you in other parts of the code. 


I forgot to say that I'm working on a better test suite for requests-oauth-hook, and some Request inconsistencies, make it very hard to write.


This will change in the next release.


This means it is fixed in version 0.7.6 ? 

Are you using my patch?


No, this will change in the next major release.
Wrong default param `config` in requests.model.Request.__init__

The default value [here](https://github.com/kennethreitz/requests/blob/develop/requests/models.py#L55) should be `dict()` as otherwise the config parsing below results in `AttributeError: 'NoneType' object has no attribute 'get'` if no config is passed.


This has bitten me today.

Kenneth, you setup `config` in `Session`. Some project might need to create `Request` objects without having to create a `Session` before. Maybe you should move `config`  setdefaults to the `Request` constructor. 

Regards,
Miguel


The config default value as dict() seems to have been fixed in [this commit](https://github.com/kennethreitz/requests/commit/35ed9b5f38ea9fa210658d555dbb8e1a71fb66d8). I believe those statements are equivalent to just having dict() as the default value, so I'm not sure why it's in this particular form.
session hooks doesn't work with v0.7.3

I was able to add a hook to a session object in v0.6.x but now it doesn't work

```
def print_url(args):
    print args['url']

session = requests.session(hooks=dict(args=print_url))
session.get('http://httpbin.org/')
```

Any ideas ? 


Yikes, sorry about that. I need to add some tests for hooks.


v0.7.4 was just released, which fixes this.


Awesome and thanks for that :)
How do I use PATCH with an external json file?

I'm currently trying to translate the example given at http://www.salesforce.com/us/developer/docs/api_rest/Content/dome_update_fields.htm into requests.  

The corresponding PATCH action is:

```
curl https://na1.salesforce.com/services/data/v20.0/sobjects/Account/001D000000INjVe  -H "Authorization: OAuth token" -H "X-PrettyPrint:1" -H "Content-Type: application/json" -d @patchaccount.json -X PATCH
```

With json:

```
{
    "BillingCity" : "San Francisco"
}
```

My current code looks like:

```
import requests

data = {'BillingCity': 'San Francisco'}
url = 'https://na1.salesforce.com/services/data/v20.0/sobjects/Account/001D000000INjVe'
headers = {
    'Authorization': 'OAuth token',
    'Content-Type': 'application/json',
}
r = requests.patch(url, headers=headers, data=data)
```

Apparently, I'm missing something from the above.  Is there a way to use PATCH in requests without writing the json content in an external file?


Oops, was using older version of requests from pypi, bug was fixed in latest version.


Sorry for the delayed response! Glad you got it figured out :)
proxychains + (socks & https proxy support)

I'd quite like to see Requests have support for proxy chaining. The actual code is pretty straight forward from what I can make of this implementation: https://github.com/pagekite/PySocksipyChain/

The httplib2 module supports both SOCKS and HTTPS proxies via the same terrible library (sockipy, as I recall). It is very frustrating to work with, so I'd recommend clean implementations of SOCKS and HTTPS proxy support.

So if you can, supporting more than a single instance of a single proxy+protocol would be great. thanks,

gq


While this is quite an edge case, I'd love to see this implemented myself. 

I have to wonder, though  —  why not just set up a local proxy that does the chaining for you?


I think I'm going to close this (for now) for being out of scope. 

Unless someone else implements, of course. Otherwise, I have no desire to do so :)


The local proxy option is a reasonable alternative, but not always practical. If I get around to requiring this functionality again, I'll do the change myself and send a pull request. 

cheers.


That would be quite welcome! (assuming it isn't too difficult to maintain)


from what I've seen, it is basically just iterating over a list of proxies (e.g. (proto, addr, port)) and issuing the correct "proxy_connect_%s'%proto call as you go. At a wild guess (ignoring the actual proxy connection code) something like 10-15 lines diff from now? ... 

the proxy connection code is more problematic as it would require either writing SOCKS4/5 + HTTPS proxy support, or trying to find a usable 3rd party module as a dependency. I know of know usable 3rd party modules. (none that I'd want near my code anyway ;)


I'm gonna be cheeky and leave this here, just so I can find it later should I return to this (or if someone else is inspired to try it, heres a reference implementation):

https://github.com/pagekite/PySocksipyChain/blob/master/sockschain/__init__.py#L923
Request.cookiejar missing on 0.7.3

This script works well on 0.6.4: https://gist.github.com/1308823

But it raises the following error when I use 0.7.3, :

```
  File "gistfile1.py", line 15, in 
  opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(response.request.cookiejar))
 AttributeError: 'Request' object has no attribute 'cookiejar'
```

I guess the API was changed but the documentation wasn't. 


Correct — a blog post and documentation update is forthcoming. CookieJars are being deprecated throughout for simpler cookie dicts.


Alright, but will it be possible to access the cookiejar object even if it is protected? 

As you can see in my app, there are some scenarios that'd be interesting if I have access to the cookiejar.


Unfortunately, no. I'm moving completely off of urllib2, which is needed for cookiejars. I might try to make a helper function available for converting the header to a cookiejar for you though.


Oh, do not worry about that. I asked that just for convenience. I'll open a pull request this change bother me a lot ;)
Detach from urllib2

- Remove auth handlers — create headers by hand.


Gah, I love this so much.


Looks like this:

https://github.com/kennethreitz/requests/commit/340f7a0733b6b023c2bec18bb0f91e8f1052a84d#L0R18

will fix an issue I hit last week:

https://gist.github.com/1292957

Good stuff!


\o/ 

urllib2 is toxic.


Trying to implement digest auth now. Easier said than done.


``` python
import requests

s = requests.session(auth=('digest', 'user', 'passwd'))
r = s.get('http://httpbin.org/digest-auth/auth/user/passwd')

```

``` pycon
>>> r.status_code
200
```
Feature/isolated unit tests

It took me a while to integrate the changes in develop (waited too long). But I created this new feature branch now which I will use to work on the unit tests. So might be best to leave it open until I have reached a good level of coverage.


Heads up — you should be writing these against the `feature/rewrite` branch — a lot of these methods won't be there anymore.


Yeah, I thought so. I looked at the branch some days ago and decided to wait a bit before I use it as the basis. Just to make sure the changes are somewhat stable. But I will base the tests on it then.

On Nov 5, 2011, at 4:31 AM, Kenneth Reitz reply@reply.github.com wrote:

> Heads up — you should be writing these against the `feature/rewrite` branch — a lot of these methods won't be there anymore.
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/kennethreitz/requests/pull/215#issuecomment-2638761


The coverage is at almost 70% now. It might take more time to push the coverage for the remaining paths. Therefore I think it might make sense to integrate the tests and extend them during actual development. This way the code can be tested better and the larger missing paths can still be added via individual pull requests. Any thoughts on this?


Any thoughts on this? I didn't have a lot of time in the last weeks so coverage is probably way lower than before, but it might still make sense to merge and continue from there.


This is really great stuff, but I worry that it'll be a maintenance liability. 


How so? They are just unit tests for the existing functionality. Or do you mean unit tests like that in general?


Yeah, I guess I'm just a fan of integration tests?


Sure seems like it :). I'm totally fine with not integrating the unit tests, if you think it's too much overhead.


For now, yes. :(
AuthManager isn't threadsafe for requests with same URL but different credentials

When you send multiple concurrent requests (using gevent, in my case) to the same URL but with different auth parameters, every request will use the same auth parameters.

Reproduced with this code: https://gist.github.com/1300341

The sample code is using the regular api, but the bug is reproducible using requests.async as well. 


Sorry for the bug. Luckily, auth manager will be removed soon. It's my least favorite part of the codebase.


This bug is fixed in 0.7.3. Thanks kenneth :)


I was so happy to see it die in a fire :)

---

:sparkles: :cake: :sparkles:
getcwd() Called on Import

Is this by design?

``` bash
sh: getcwd() failed: No such file or directory
sh: getcwd() failed: No such file or directory
```


Are you sure something else isn't causing this?


I'm pretty sure _something_ inside requests is causing it.  Basically this occurs within a directory that no longer exists (or whenever getcwd() would fail).

Example imports:

``` bash
wolf@gs9671:/tmp/src$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> exit()
wolf@gs9671:/tmp/src$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
sh: getcwd() failed: No such file or directory
sh: getcwd() failed: No such file or directory
>>> 
```


I'm just more wondering why getcwd() is needed, not that it always happens.  Sorry should have specified that!


It's not needed, and it's not called explicitly — that's why I'm confused :)


I thought it might be for including a non-standard library, seems not:

``` bash
wolf@gs9671:/tmp/test$ python
Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import flask
>>> import requests
sh: getcwd() failed: No such file or directory
sh: getcwd() failed: No such file or directory
>>> import redis
>>> import bitarray
>>> import termcolor
>>> import gdata
>>> import pygit2
>>> 
```

Those libraries are installed in /usr/local/lib/python2.6/dist-packages/*

Anyways, just an oddity, not a show-stopper.


That is beyond weird. I have no idea. Let me know if you come up with anything.
Syntax Error in requests/async.py in Python 2.5

Function calls like `func(*args, arg=val, **kwargs)` raise a SyntaxError in Python 2.5 (tested in both 2.5.2 and 2.5.6)

I have a patch for the issue here:
https://github.com/stoey/requests/commit/84e89429f0414fe0d6b9497b16833bc6bee48957

Let me know if you would prefer a pull request of that commit.  

The rest of my changes are working around a bug in the urllib2 in Python 2.5.2  that was fixed in 2.5.4 so are not very useful to anyone else.

Cheers,
Dan


Sorry for the bug! A pull request would be awesome :)
Added support for Eventlet to async module

A new setting is added: `async_module`, which defaults to 'gevent'. If gevent is not found eventlet is tried in turn, and if 'eventlet' is set explicitly, gevent will not be used.

Example to use eventlet:

```
from requests.config import settings
settings.async_module = 'eventlet'
from requests import async
....
```


Thanks a ton for this! Very exciting ;)

I'll try to push this out in a release this evening.


From http://www.gevent.org/

> gevent is inspired by eventlet but features more consistent API, simpler implementation and better performance.

Do we really need to support eventlet considering the above?


No. The only reason to have Eventlet support is because it's pure-python, and therefore works in PyPy.


I'd like to add it in as a fallback, but not a configurable one.


Would also allow us to use requests on stackless which gevent doesn't support yet.

Any progress on this?
[API review needed] Added support for Eventlet to async

Hi,

Since Gevent and Eventlet are similar I thought I'd add support for the latter :-)

Since both need monkeypatching things can get a bit tricky. Please see my proposal in the pull request commit.

With this approach someone which would like to use Gevent would do:

`from requests.async import gevent as async`

and someone who want to use Eventlet would do:

`from requests.async import eventlet as async`

How does it sound look to you? Any suggestion?

If you find it ok I'll complete the pull request with required changes to documentation.

Cheers,


Thanks for the patch! 

I love the idea of supporting both, but it should be a transparent thing. Here's the workflow that I have in mind:

### Workflow
- Import gevent
- If ImportError on gevent, import eventlet
- if no eventlet or gevent, raise existing warning that requests requires gevent.

The api should be identical to what it is now :)


I tried that first, but I thought that it's be better to let user choose, because you may have both installed but you'd like to use just one. Imagine somebody develops an application which uses requests with Eventlet and does not use Gevent at all. I guess it would like to use Eventlet no matter what the user has installed in his system.

How about this: Maintain the API and use your workflow, _but_ if the REQUESTS_ASYNC_MODULE environment variable is defined we _only_ try that module. It should be either 'gevent' or 'eventlet'. This way there are means for the user to decide which one to use.

Does it sound better?


I'm not sure why anyone would ever use eventlet if gevent was available, but I'll go with it :)

Sounds like a plan. I maintain a set of module-level defaults too that can be overridden in `requests.config`. You can just set it there as `settings['async_module'] = 'gevent'`, and someone can change that if they so choose.


Ah, I missed `requests.config`. I'll add it there, and then there should be no fallback. If someone wants Eventlet support it needs to explicitly 'ask for it'. Gevent would be the default.

I'll make a new patch.


Essentially I was planning on making Eventlet a fallback for when Gevent wasn't available. I still want this to be the case

Adding the overridable default will give someone a little more control if they need it.


If gevent fails to import, give Eventlet a shot. Our usage is simple enough that it doesn't have to be explicit.


I fucked up my repo and made a new one. GitHub won't allow me to attach commits from other branches to this pull request, so I'm closing it and opening a new one. Sorry about that :-S


#207
allow_redirects=False ignored in a POST request

Hi,

I just noticed that if you will send a POST data into a URL which will redirect you then allow_redirects = False are ignored

``` python
d_args = {'a':'a'}
r = requests.post(url = 'http://www.synopsi.com', data = d_args, allow_redirects = False)
print r.url => 'https://www.synopsi.tv/'

d_args = {}
r = requests.post(url = 'http://www.synopsi.com', data = d_args, allow_redirects = False)
print r.url => 'http://www.synopsi.com/'
```


Fixed the in the upcoming release!
Support to Disallow fetching if content size large

Allow support in request to disallow fetching a url, if content size is above a certain size specified in the argument list. This is very useful in cases where one needs to disallow large requests.


Actually, when you make an http request, the content isn't downloaded until you access the `Response.content` attribute :)


So how would we check the size of the response? 


The headers, if they are available. The raw socket is also available from `Response.raw` in the latest release.


Ok. Are their some more docs planned to show off usage?


Yes :) 

#178


Thnx
Beautiful Async Module

``` python
# -*- coding: utf-8 -*-

import requests
from requests import async

urls = [
    'http://www.readability.com',
    'http://tablib.org',
    'http://httpbin.org',
    'http://python-requests.org',
    'http://kennethreitz.com'
]

urls = urls + urls

```

**Standard (14.125s)**

``` python
for url in urls:
     r = requests.get(url)
     r.content
```

**Standard + Gevent (1.589s)**

``` pycon
>>> rs = [async.get(u) for u in urls]
>>> async.map(rs)
[<Response [200]>, <Response [200]>, ...]
```


Hmm, I think that's about it.
matrix parameters in an earlier segment get URI encoded?

It looks as though matrix parameters that are in a non-leaf segment of the URI path get URI encoded and so corrupted.

E.g.

http://server:port/root;property1=value1/path/name.type

becomes

http://server:port/root%3Bproperty1%3Dvalue1/path/name.type

but if you put them at the end it is OK:

http://server:port/root/path/name.type;property1=value1

__Jason


Is this still an issue?

Unfortunately, matrix parameters aren't supported at this time. But, they shouldn't be meddled with in the first place.


Closing for now — please comment if this is still an issue.


It would be an issue if I had to use this framework. I was considering it as an alternative to restful_lib, but the matrix parameter issue is a blocker.
Fix errors introduced by merging an outdated pull request.

See issue #195.

(Note: you can still specify an iterable of functions as a hook, no worries.)


The pull request was not merged unintentionally. I'll merge this in, but only temporarily. 

Clearly I should be doing this rewrite in a separate branch.


I'm pretty sure you did it intentionally, because it added some functionality. But the pull request you merged was outdated, and thus no more compatible. I tried to retain new functionality, as you can see by comparing hooks.py. Please pay more attention to merge conflicts, OK?


I'm extremely aware of everything the pull request did. It was under debate for over a month.

If you are going to contribute to this project, please do so respectfully.


Well then, maybe I understood something wrong. Sorry.
Now it seems completely broken.

That was a good day, and I was trying to retrieve the response content, when suddenly a wild exception appears.

``` python
>>> import requests
>>> response = requests.get("http://httpbin.herokuapp.com/")
>>> response.content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests\models.py", line 458, in content
    self._content = self.fo.read()
AttributeError: 'Response' object has no attribute 'fo'
```

I tried to fix it by replacing line 162 of models.py (`response.raw = resp`) with `response.raw = response.fo = resp`, but I got the following error:

``` python
>>> import requests
>>> response = requests.get("http://httpbin.herokuapp.com/")
>>> response.content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests\models.py", line 481, in content
    if settings.decode_unicode:
NameError: global name 'settings' is not defined
```

Hmm... `from .config import settings`?

``` python
>>> import requests
>>> response = requests.get("http://httpbin.herokuapp.com/")
>>> response.content
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests\models.py", line 493, in content
    self._content = get_unicode_from_response(self)
  File "requests\utils.py", line 291, in get_unicode_from_response
    return unicode(r.content, encoding)
  File "requests\models.py", line 493, in content
    self._content = get_unicode_from_response(self)
  ...
RuntimeError: maximum recursion depth exceeded
```

Now I'm pretty sure I can't fix that. What is happening?

If that helps, everything was OK in [commit f524fe4](https://github.com/kennethreitz/requests/commit/f524fe4934d41fc08173cd61f49c385207d42390). This problem probably exists because you merged an outdated pull request.


I've created a pull request. This issue can be closed now.


Please do not use the current develop branch. It is under _heavy_ development, and is not suitable for anything at this time.


I _don't_ use requests in my production code at all, honestly. ('Cause I don't write any production-level code.) And I like to use all the newest and most broken stuff I can get :-)


v0.6.2 is fully suitable for production, as are all releases. :)
Broken link in docs: "supported interpreters"

Hi,

The "supported interpreters" link in the docs is broken:
http://docs.python-requests.org/en/latest/community/faq/#python-3-support


< kennethreitz> blame readthedocs :)


I’m not sure I understand. The `faq.rst` file contains the following markup:

```
Here's a list of `supported interpreters <interpreters>`_.
```

which, unsurprisingly generates the following HTML:

```
Here&#8217;s a list of <a class="reference external" href="interpreters">supported interpreters</a>.
```

But I can’t find a `interpreters.rst` file anywhere in the repository. How is that related to readthedocs? Maybe Kenneth forgot to commit a file in 47a36de?


I don't understand either - I pointed it out to him on IRC earlier today, that was his response; I'm just passing it along.


Looks like I spoke too soon :)


Fixed!
Python 3: fucking accomplished.

Hi!

I like Requests very much, but Python 3 is too much awesomeness to miss. Yeah, I know, you're working on python 3 support but I live in Russia and "I'm working on it" actually means "fuck you, I don't care about your needs" for me :-)

So a few hours ago I <del>drank some vodka and</del> decided to rewrite Requests with Python 3 support by myself. I got a bit carried away in the process, though, so the result does not look like Requests very much, but it still has almost the same API. (Some things like `Request.cookies` and `Response.cookies` are missing, but I'll fix it. Maybe.)

Here it is: [pyos/requests3](http://github.com/pyos/requests3). Could you take a look at it, please? I'd really appreciate some thoughts on my code. ^^

(Not sure if creating an issue is a right thing to do to tell someone about that thing; I'm sorry if I made something wrong :-)


Thank you very much sir!

I can assure you, Python 3 is the very next thing on the roadmap. I'm currently reimplementing all of requests to be powered by urllib3 instead of urllib2, which will make this port significantly easier — so, if you can be patient for a few more weeks, all will be well :)

There's really no need to reimplement the entire library to support Python3. It'll actually be quite a simple process as soon as this dev cycle is finished :D


> There's really no need to reimplement the entire library to support Python3.

To be honest, I've done this not only because I like to code stuff in Python 3, but also because I didn't understand most of the code.

For example:
- Why do you need hooks when you can modify arguments before calling `request()` (this will also modify the `Request` object) and the `Response` object after `request()` finishes?
- Why bother encoding keys & stuff in `utils.encode_params` if `urllib.urlencode` does the same thing for you?
- Why copy the scheme by yourself in `utils.cleanup_url` when `urlparse.urljoin` also does this?
- What exactly urllib3 does other than maintaining a collection of HTTPConnections and encoding data in `multipart/form-data`?


- Hooks allow you to do some pretty advanced things, like add simple OAuth support.
- You're welcome to call a few extra functions _every time_ you want to send a parameter.
- See above.
- That's mostly it.
No data parameter in requests.delete()

The requests.delete() method does not support having a `data` parameter. The [HTTP spec for DELETE](http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html) does not give much guidance about whether this is allowed, but in practical situations, it is often used. There is currently no way to access such services using requests.

(In particular it's worth noting that `curl -X DELETE -d "data"` works without issue)


According to the HTTP spec, "The DELETE method requests that the origin server delete the resource identified by the Request-URI."

This would seem to negate the usefulness of passing data in the request body, as the resource should be solely identified by the Request-URI.


I agree that a theoretical discussion can be had about this, and I'd probably be on your side, @sergedomk. However, we cannot ignore that, _in reality_ servers can and do use data in the request body even for DELETE requests, and other standard tools (like `curl`) support this.

If you want an example, here's the first one that comes to mind: http://docs.gnip.com/w/page/23733233/Rules%20Methods%20Documentation#deleterules


I'd say this is a point for theoretical discussion.

Submitting a DELETE with data is already possible using `requests`.

`requests.request('DELETE', url, data="data")`

So the discussion is whether or not it should be included in the DELETE method as well. It's a question of where to draw a boundary on adhering to the spec vs. not.

`curl` is a great example of this as it has no adherence to the HTTP spec at all. `curl -V FOOBAR -d "data"` also works perfectly fine. And FOOBAR is definitely not a HTTP method.


Oh, interesting. I didn't know about `requests.request`. That certainly mitigates my issue, and makes me agree with you even more.

I'm okay with closing this issue if @kennethreitz is.
Make get and head requests respect allow_redirects=False.

Fixes issue #183

I did this on top of the 0.6.1 release because I need to use it in production.


Just realized I forgot to add myself to the authors file... Would appreciate if you did that if/when you merge this.


You can still add yourself by commiting the change. It will be added to the pull request


Thanks for the pull request! This was actually fixed a few weeks ago for the rewrite (currently being done in develop).

I'll put out a bugfix release in the meantime.


Merged. Thanks!

:cake:
Cannot prevent redirect on GET request

I would like to make a request, but I do not want to follow the redirect. Currently this does not seem to be possible. I get a 200 response code despite initiating the request with `allow_redirects=False`. Here's some example code to show what I'm doing:

``` python
response = requests.get('http://www.test.com/redirect', allow_redirects=False)
assert response.status_code == 302
```

Upon investigating the source, it seems that `allow_redirects` is ignored for GET and HEAD requests.

I'm using v0.6.1


See #184
Response URLs are not unicode strings

When running...

```
>>> u = requests.head('http://t.co/qewwD8xr').url
>>> print repr(u)
'http://korrdon.info/night/music/32843-fedde-le-grand-live-space-ibiza-closing-party-2011-02102011.html?utm_campaign=\xd0\x9a\xd0\xbe\xd1\x80\xd1\x80\xd0\xb5\xd1\x81\xd0\xbf\xd0\xbe\xd0\xbd\xd0\xb4\xd0\xb5\xd0\xbd\xd1\x82+\xd0\x94\xd0\xbe\xd0\xbd\xd0\xb5\xd1\x86\xd0\xba\xd0\xb0&utm_term=KorrDon&utm_content=news&utm_source=korrdon.info&utm_medium=korrdon'
```

... I had expected `u` to be equal to:

```
u'http://korrdon.info/night/music/32843-fedde-le-grand-live-space-ibiza-closing-party-2011-02102011.html?utm_campaign=\u041a\u043e\u0440\u0440\u0435\u0441\u043f\u043e\u043d\u0434\u0435\u043d\u0442+\u0414\u043e\u043d\u0435\u0446\u043a\u0430&utm_term=KorrDon&utm_content=news&utm_source=korrdon.info&utm_medium=korrdon'
```

That is, `u.decode('utf8')`.


Have you taken into account charset returned by server?

```
>>> requests.head('http://t.co/qewwD8xr').headers['content-type']
'text/html; charset=WINDOWS-1251'
```


They definitely should be.
Append '/' to domain root requests


Might be a better idea to strip the domain rather than appending a character. At least it'll be more inline with other major libraries like cURL.

Sucks to make a special case just for Twitter though. :/


You know, we should have done like a mini-sprint this weekend. :-/

---

For now, I'm appending `/`, because I haven't taken a look at the the proxy stuff yet.