Raibek/requests-github-issues huggingface dataset:
Give Response objects a length, and make them iterable

I've often it useful to be able to call len() and iterate (by line) over Response objects. This pull requests adds the `__len__()` and `__iter__()` Python special methods to the Response class.


These things have both been considered and proposed before, and we have opted out for various reasons.

Thanks for the contribution, though! It's a logical change, just not one that we're ready to make yet :)


@mwielgoszewski just to elaborate on @kennethreitz's comment.

With the `pre_fetch=False` option, your solution will not work for `__len__` when the server doesn't provide a Content-Length header. As for iterating over the response, there's far more to a response than just the content. So iterating over a response makes little sense in my personal opinion. As it is, the API for iterating over the content is complete for both values of `pre_fetch`.

Stick around, I'm sure there's far more you can contribute elsewhere in the project.
Unicode filename uploading

When trying to upload filename with non ASCII symbols (for example: "Тестовый файл.txt") I'm getting this exception:

> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 60: ordinal not in range(128)

This commit fixes this. 


Hi @ekratskih, as you might be able to tell, this is a problem with urllib3 (which I maintain). I welcome you to submit a pull request with a similar fix directly to urllib3, but please include a unit test. https://github.com/shazow/urllib3


Thanks for the report @ekratskih, and thanks for taking this off our hands @shazow! :cake:

Closed as 'not our fault'.


Related: shazow/urllib3/issues/119
"Content-Length": 0 Doesn't Work

Hi Everyone,

This is rather a minor issue. However it makes more sense if integer is supported.


We've discussed this in the past (I cannot seem to find the issue though). Integers don't actually make sense in the context of the specification, if I remember correctly.

--EDIT-- 

Found it: #865


I see, had a misunderstanding of Content-Length. It's actually in octets, and the simplest representation of octets in Python are in strings.

Closed (hope this helps other ppl who wonders about similar issues)
Improved conformance of link header parsing. Added unit tests as well

In [RFC-5988](http://tools.ietf.org/html/rfc5988) link headers are allowed to specify multiple rels per link by separating them with a space

```
Link: </>;rel="rel1 rel2"
```

In addition, there can be multiple links with the same rel, which the current .links() implementation doesn't allow since it overwrites earlier links with later ones.

This patch adds both of those capabilities, and in addition changes the "url" tag to the "href" tag in order to align better with the RFCs terminology (because the link header derives from the link html tag, which uses href).

Several of the regex's used are modified version of those from the mnot/redbot project.

I added unit tests in the patch, but there are also some basic end to end tests [here](https://gist.github.com/4191608)


Thanks for the pull request!

Unfortunately, I won't be merging this at this time. All of the fallacies that you pointed out were known at the time of implementation. We decided to go the pragmatic route :)

In the future, we may utilize a MultiDict to have multiple values for a single key.


So, is the main issue with this patch is that it returns a dict of lists, rather than a multidict?


I'd say so.

It's also significantly harder to maintain and complicated, but the MultiDict is the main thing.
Add the ability to turn off HTTP 0.9 support.

While debugging an issue I discovered requests was coming back with 200
response, when it really shouldn't have.  It turns out this happened for
two reasons: the jetty server running the app was rather lame and
didn't fail the request as a Bad Request, and httplib was happy to let
malformed data through and call it success.

It turns out httplib's strict flag controls this behavior of whether or
not to validate the status line.  The underlying urllib3 supports the
concept as well.  There was a bug there to that is now fixed upstream.

The last step is exposing this through requests.  This introduces a
supports_http0.9 flag to help control this behavior.  It defaults to
to True to preserve the current behavior.   Setting it to False will
allow the underlying HTTPConnection to validate the status line.


FYI: I did try to add a test for that, but didn't find a publicly available service to test against.  I could create a test that stands up a couple of sockets, if you like.  But I wanted to see what you guys thought before going through that trouble.

Also, this will require a new urllib3 to make this work, since urllib3 was not passing the `strict` flag to the underlying connection.  The flag was already present in the API (so this landing on master won't cause a problem)... it's just that strict mode won't actually be enabled without a newer version.


Thanks! I'll be happy to add this if you remove the argument. I don't see any reason to make this optional.


Actually, I'll merge it now. It may get changed before release though :D


I didn't see any other way for a user to pass a config in to override the default.  Am I missing something?


BTW, thanks for merging it!


absolutely!
Added support for persistent connections (keep-alive) in proxy mode

Fix for issue https://github.com/kennethreitz/requests/issues/875
- Proxy is now also comes from the connection pool if keep-alive is enabled
- Also contains tests to see if a connection is pooled or not


Thanks! I love the simplicity of this too. Much better approach.

:sparkles: :cake: :sparkles:


@vlaci 
Have you seen https://github.com/shazow/urllib3/blob/master/dummyserver/server.py?


I hadn't but after I was I'd say the wsgi implementation is not applicable here. In fact, tt may have been worth using the SocketServerThread one if it was included in requests' repository but it wasn't.
JSON request support

Hi Kenneth,

I know you remain unconvinced about https://github.com/kennethreitz/requests/issues/769 etc., but I keep wanting to make my code simpler by writing this

```
requests.post(url, json={...}).json
```

And I suspect I'm very much not alone. I'd like to start by maintaining a fork that is at least reasonably correct, if not good enough for inclusion. So, if you could poke some holes in my first stab at implementing this, I would appreciate it.

Thanks.


Try writing this as a hook. This way you don't need to fork anything :)


Check out the `magic_json` branch on my github3.py project. The hook @piotr-dobrogost mentioned can be seen in action there and is written for my needs.


Also, had you bothered to search through the closed issues, your suspicion would be confirmed and you would also know that it has been denied. It isn't the role of requests to duplicate what a single extra import and call would do. And when written not as a hook, it will provide a shorter stack trace when you pass improper JSON.


@sigmavirus24, he referenced #769 in his post, he knows. =)


Apologies. This is what I get for reading it on a "smart"phone screen. The 
GitHub for Android app seems to spuriously not show links.


not happening ;)
quickstart and api docs: prefetch=False is required to read raw content

Quickstart section "raw response content" does not say about the requirement: https://github.com/kennethreitz/requests/blob/master/docs/user/quickstart.rst#raw-response-content

Also API docs at http://docs.python-requests.org/en/latest/api/ missing this requirement

(requirement introduced on 0.13.6)


Resolved by #987. Thanks for the report!
Build issue related to s/chardet/charade/ change

Hi, I'm seeing this in my 'pip install' command:

```
Obtaining requests from git+git://github.com/kennethreitz/requests.git#egg=requests (from -r requirements.txt (line 19))
  Cloning git://github.com/kennethreitz/requests.git to ./env/src/requests
  Running setup.py egg_info for package requests
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/var/local/maria/hudson/workspace/platform-api-console/env/src/requests/setup.py", line 6, in <module>
        import requests
      File "requests/__init__.py", line 53, in <module>
        from .models import Request, Response
      File "requests/models.py", line 37, in <module>
        from .compat import (
    ImportError: cannot import name chardet
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/var/local/maria/hudson/workspace/platform-api-console/env/src/requests/setup.py", line 6, in <module>

    import requests

  File "requests/__init__.py", line 53, in <module>

    from .models import Request, Response

  File "requests/models.py", line 37, in <module>

    from .compat import (

ImportError: cannot import name chardet
```

Pinning to 8f86ff633a2646442f2d8976046b5c0025e772c1 works around the issue.


Yes, it's under development and is not working properly at this time.


I also have this error. Has this been fixed?


Yeah, this was fixed a very long time ago. Are you sure it's exactly this error?


Thanks for the quick response. I'm getting this error after I use py2exe on my app that uses requests. My apologies if this is a different error.

  File "requests__init__.pyc", line 58, in <module>
  File "requests\utils.pyc", line 25, in <module>
  File "requests\compat.pyc", line 7, in <module>
ImportError: cannot import name chardet


Yeah, that will be a different error. py2exe is fiddly when it comes to packaging. You'll want to take a look at what `setup.py` includes in the [packages list](https://github.com/kennethreitz/requests/blob/master/setup.py#L17-L25), and make sure that all of these folders end up present in your py2exe build.


Yeah, I added the packages but when I try to compile, I get "error: package directory 'requests' does not exist". So finicky. Do I need to put 'requests-2.2.1-py2.7.egg' instead of 'requests' or is there are a parameter to specify my site-packages folder, even though I couldn't find it when I read the documentation.

Update: I changed it to only include the package 'requests' instead of all those that were listed, and changed the syntax to match the syntax on the py2exe site and it seems to work right now. I think the info in that link is wrong or outdated.


Uh...the info in what link?


The link you gave in the previous reply. :)


Oh, I see. No, that's not out of date, it just serves a specific purpose. py2exe almost certainly uses a totally different mechanism, meaning that what you do with that is likely to be different.
Auth persists though redirect.

After the refactor is finished, we should take a look at the default Auth handlers. Right now their authentication information will be passed on a redirect, which is not forbidden by the spec, but is probably not the best default behaviour. We should probably be secure by default.

This behaviour should probably be optional, in case anyone relies on it.

This is related to and inspired by #627.

This should not be started until after #970, i.e. after v1.0.0.


I feel like these can be optional arguments to the author handlers.

Sent from my iPhone

On Nov 28, 2012, at 1:49 AM, Cory Benfield notifications@github.com wrote:

After the refactor is finished, we should take a look at the default Auth
handlers. Right now their authentication information will be passed on a
redirect, which is not forbidden by the spec, but is probably not the best
default behaviour. We should probably be secure by default.

This behaviour should probably be optional, in case anyone relies on it.

This is related to and inspired by
#627https://github.com/kennethreitz/requests/issues/627
.

This should not be started until after
#970https://github.com/kennethreitz/requests/issues/970,
i.e. after v1.0.0.

—
Reply to this email directly or view it on
GitHubhttps://github.com/kennethreitz/requests/issues/975.


+1, that was my intended design too. Not gonna touch it though until you're done, so that I don't rely on an API you change. =D


"This behaviour should probably be optional, in case anyone relies on it."

I am relying on it, and hope to figure out a way to get the same effect in urllib2; if someone happens to know how, feel free to answer the related [stackoverflow question](http://stackoverflow.com/questions/13701082/urllib2-http-basic-auth-after-redirect).


Touching back on the main topic, how would the authentication handlers handle this? Would they have access to the url being requested and only perform authentication for a specified domain (if an option to do so is set)? 


@lukasa I am holding your feet to the fire on this one. ;)


@sigmavirus24: I'd forgotten about this, so thankyou! I'll take a look at it tonight and see how problematic it's likely to be.


Should auth persist across redirects to subdomains? What about from one subdomain to another? Put another way, should auth persist in these situations (not likely to happen in real life)?

http://google.com/ -> http://mail.google.com/
http://www.google.com/ -> http://mail.google.com/

Thoughts @sigmavirus24 @kennethreitz?


We should keep it simple.


if someone doesn't want it to persist through redirects, then they shouldn't allow redirects.


@kennethreitz, while I like that I think if we see a 307 (after a POST) we should just return that response as if the user had specified allow redirects to be False. It's safer and it follows spec.


@kennethreitz; Are you happy to make that the policy? If so, I'm happy to simply say that the user should disallow redirects and not write any code. =P


Yes :)


Heh, awesome, then I pronounce this issue closed as 'Not a bug'. =D


Oh god, I keep confusing issues.
Delete cookie

Does requests support cookie deletion?  Here is a simplified example trying to communicate with Tornado

On the server side...

```
class Login(tornado.web.RequestHandler):
    def post(self):
        self.set_secure_cookie('id', 1234)

class Logout(tornado.web.RequestHandler):
    def get(self):
        self.clear_cookie('id')
```

On the client side doing something like this...

```
cookies = {}
session = requests.session(cookies=cookies)
session.post('http://localhost:8080/login')
print session.cookie['id']
session.get('http://localhost:8080/logout')
print session.cookie['id']
```


Yeah. `session.cookies` should be an instance of `RequestsCookieJar` [https://github.com/kennethreitz/requests/blob/master/requests/cookies.py], which implements `MutableMapping` and should allow deletion through the dictionary interface, e.g., with `del`.


Wait, I misunderstood you --- the question is whether we correctly implement deletion of cookies at the server's request. The convention for doing this is for the server to send a Set-Cookie header with an expiration time in the past; see page 7 of [http://tools.ietf.org/html/rfc6265].

Our management of this is deferred to Python's `cookielib`, but AFAIK it is working --- is it not working for you? If not, could you write a test case against httpbin.org?


We use the standard library's cookie handling code, so if it does it right, so do we. =)
Can't pip install f553857beb4424786f8d665019686551905483d6 - error: package directory 'requests/packages/chardetrequests/packages/urllib3' does not exist

```
mlm@matt-mmf-macbook.local:~
$ pip install git+https://github.com/kennethreitz/requests.git@f553857beb4424786f8d665019686551905483d6#egg=requests-f553857
Downloading/unpacking requests-f553857 from git+https://github.com/kennethreitz/requests.git@f553857beb4424786f8d665019686551905483d6#egg=requests-f553857
  Cloning https://github.com/kennethreitz/requests.git (to f553857beb4424786f8d665019686551905483d6) to /var/folders/vk/p95wm1qx0vdf0087p0z9qzc40000gn/T/pip-build/requests-f553857
  Could not find a tag or branch 'f553857beb4424786f8d665019686551905483d6', assuming commit.
  Running setup.py egg_info for package requests-f553857

    error: package directory 'requests/packages/chardetrequests/packages/urllib3' does not exist
    Complete output from command python setup.py egg_info:
    running egg_info

creating pip-egg-info/requests.egg-info

writing pip-egg-info/requests.egg-info/PKG-INFO

writing top-level names to pip-egg-info/requests.egg-info/top_level.txt

writing dependency_links to pip-egg-info/requests.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/requests.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found



error: package directory 'requests/packages/chardetrequests/packages/urllib3' does not exist

----------------------------------------
Command python setup.py egg_info failed with error code 1 in /var/folders/vk/p95wm1qx0vdf0087p0z9qzc40000gn/T/pip-build/requests-f553857
Storing complete log in /Users/mlm/.pip/pip.log
```


@matthewlmcclure try out my branch, this should fix your issue.
Real logging

- WARNING for things that shouldn't happen and can cause unexpected behaviour or problems, 
- ERROR for when things are broken, 
- INFO for major operations (i aim for about 1 info message per major event, like a new connection), 
- DEBUG should have enough information to debug WARNINGs


https://github.com/shazow/urllib3/blob/master/urllib3/__init__.py#L29


the "right" thing to do is only add handlers to the log namespace you're interested, ie.

correct: `logging.getLogger('myapp').addHandler(...)`
wrong: `logging.getLogger().addHandler(...)`


I'd like to add that any time there's a log statement within the beginning of an except block, that `log.exception()` is called to capture the backtrace along with the error message.


Need to document.
Refactor

Breaking changes. Fixing things. Making this project sustainable. 
## Goals
- Externalize functionality, utilize new Requests organization. (#938)
- Introduce better object model (https://twitter.com/dozba/status/273477717721223168)
- Introduce Transport Adapters (#895)
- Remove most configuration.


Do we need safe mode? Seems silly.


It's a maintainability headache: chuck it. =D


Also removing danger mode.


There shall be only one mode: awesome mode.


Strict mode is valid, but only if we plan to support other RFC-complaint behavior. Currently, this is only POST redirects. 

Any others in the foreseeable future?


Maybe modes should be accomplished via hooks, which could be maintained as a separate project like Piotr suggested.


Maybe.


Hand written notes about the refactor:

![notes](http://cl.ly/image/1u0h0v2Y3X3X/ClouDrop%2520Nov%252027,%25202012,%252012:03:09%2520PM.jpeg)


`Request` -> `PreparedRequest` -> `Response`


`Request` -> `PreparedRequest` -> `Session` - > `TransportAdapter` -> `Response`


Does a session send a request, or does a request send itself through a session?

I think a session sends a request through a connection.


`Request.send()` always felt a bit weird, idk.


You can create a request yourself, send it through a session. No reason to use connections directly.

api/session level stuff will remain as it is today.


``` pycon
>>> Request(method=..., url=...).prepare()
<PreparedRequest [GET]>

>>> session.send(_)
<Response [200]>)
```


timeout attached to send


Hmmm, perhaps cert, verify too


yeah, that makes sense


and prefetch


Man, this is going _really_ well.


I hope so, I woke up to about a million emails. =D


http://cl.ly/image/02031C300N0h


> http://cl.ly/image/02031C300N0h

I :heart: you so much showing the body of the request.


@sigmavirus24 the best part is that it's fully mutable :)


:+1:


So close!


I think all that's left is:
- Proxies
- Session adapter registry


Seriously, so close. Just gotta trust the env now.


DONE
Switch to Apache 2

I've been wanting to do this for a long time. 

I don't care bout incompatibility with GPLv2.


Evaluate licenses of vendored libraries (NOTICES) for compatibility.


Shouldn't be a problem.


Why not MIT (or Zoidberg)?


The current license is functionally equivalent to the MIT/X11 and 2-clause BSD.

Apache 2 provides a number of protections that a project with an audience as large as Requests could benefit from.


One major disadvantage of Apache 2 is that it requires a license line to be added to each file. Super annoying, but worth it.


@kennethreitz: I can't see that requirement in the License instructions...


@Lukasa hmm, it's a long document. I might be crazy ;)


Humorously enough @kennethreitz, you might be thinking of the GPL's requirement to have itself in **every** file ;)


Haha, quite possible! 


=P Nice.

Anyway, we currently vendor urllib3, chardet, chardet2 and oauthlib. They use: MIT, LGPL, LGPL (again) and and 3-clause BSD. Those are the licenses we need to be compatible with.

The LGPLs might be...fun.


Fucking character encodings.


Indeed.

So, [it looks like](http://stackoverflow.com/questions/1174240/apache-and-lgpl-license-compatibility) we're fine on the LGPL front so long as we don't change chardet. If we fork chardet (see #951) then that would need to be relicensed under the LGPL. Of course, we could do that as part of the Requests org and then vendor that into requests as a relicensed copy.

My head hurts.

For now, we're fine on the LGPL front.


Hmm, I was planning on forking it :(


Looks like MIT and BSD are fine too.


Sweet, pulling the trigger.


@kenneth: You'd have to fork it as a separate project, relicensed under the LGPL, and then vendor that forked project into Requests. Not the end of the world, but a hassle.


@Lukasa I think it's appropriate.

What shall we call it? Chardetter? FuckUnicode?


funicode.


How about [I Don't Know Exactly How, But I'm Sure This Is Joel's Fault](http://www.joelonsoftware.com/articles/Unicode.html).


Or ForkChardet with a bit of a tongue-in-cheek meaning ;) (Lame I know, I'll see myself out)


I actually do want to call it FuckChardet, but I don't want to promote any negativity. 


Charred. 


charmander.


CharStar? Pointers and whatnot :P

@shazow I was thinking of that too ;)


crash


http://www.wolframalpha.com/input/?i=words+that+start+with+char


Charted


Oooh "Charade"


sharted.

(Also, +1 to Charade.)


Charade. Sold.
Pass hooks to redirected requests.

Basically what it says on the tin.

Discovered while providing a solution to #627.


Hmm, in the future i think these will be persisted in the session.


Agreed, it would be a better model.


They still needed to be passed though, unless I'm missing something.


In the refactor, the sessions will ideally be accessible to all the requests.


Was this removed with (or before) Requests 1.1? My hook is not getting called when following redirects. I made essentially the same change as in the commit. However, this resulted in the hook getting called twice for the final 200 response. I am preparing a pull request for Requests 1.1, that does not double-call the hook.
More complete OAuth tutorial documentation

The documentation at http://requests.readthedocs.org/en/latest/user/quickstart/#oauth-authentication explains how to use:
1. client_key
2. client_secret
3. resource_owner_key
4. resource_owner_secret

But it omits the steps to get a resource_owner_key and resource_owner_secret.

It would be nice to have a self-contained start-to-finish tutorial to make an OAuth 1.0 (RFC 5849) authorized resource request to, say, Twitter's API, and maybe one other using OAuth 2.0.


Agreed.

However, we're taking steps towards pulling the OAuth code out of Requests itself and into the new Requests org (see #938). This will involve removing the OAuth stuff from the main Requests documentation and adding it to [requests/requests-oauthlib](https://github.com/requests/requests-oauthlib). Can I ask you to raise the 'start-to-finish' tutorial idea as an issue on that repository?

Thanks for the idea, it's a good one!


@Lukasa for the time being I thinik it's fine to add it to the Requests documentation since that's where everyone will be looking for the time being.

@matthewlmcclure it won't hurt to submit a PR. On the other hand, afaik, oauthlib doesn't have OAuth2 completely finished. Their API might be finished but having information in the tutorial about that may be misleading for the moment. As for a start-to-finish tutorial using the Twitter API as an example, perhaps we could get Twython's permission to excerpt/adapt some of their code for the example?


Once we move it out, I still want the documentation to exist in Request's docs. It'll be official, just outside of core.

A bit of a refactor of the documentation is on order. I've been thinking about this lately. We'll need an 'Authentication' guide, with sections for every type of authentication.


I'm working through the process of getting it to work myself, and I left some useful notes on getting it to work in #966.


The documentation for OAuth is now quite broken. I'm unsure how to actually get it working again :-(


@r1chardj0n3s it'll be updated shortly. The new library that provides the OAuth class is `requests-oauthlib`


Thanks!
Add a bit more to the developer's documentation

I'm not sure what else could be added to satisfy #601. If this is sufficient,
I suggest closing that.

Also, this is just a docs change, if Travis fails (since it seems backed up
and I don't have the time to make sure this passes) it isn't my fault. ;)


@kennethreitz, before you merge this, can we get a consensus on whether we officially support 3.1 and 3.3? If we do, it might be worth getting TravisCI to test on them so we don't regress.


I'm fairly confident that @travisci never added python 3.3 support although I haven't received any answer either way. My reasoning for claiming the supported versions were up to and including 3.3 was [setup.py](https://github.com/kennethreitz/requests/blob/develop/setup.py#L67). If that is wrong, I don't want to be right ;)


Mm, I know that Kenneth wants to support 3.3, with 3.2 and 3.1 being bonuses, but if we're going to claim we support something we should make sure not to regress on it. =) Otherwise, we shouldn't claim we support it so that people don't get mad when it breaks.


@Lukasa I should have added that I'm +10 on waiting for @kennethreitz to weigh in. ;)


Perfect :)

---

We'll drop 3.1 support when Travis drops support for it, which is soon.
New project requests-hooks?

@kennethreitz 

How about creating new project requests-hooks with various hooks for requests? I could maintain it if you want.


I'm +1 many times over. ;)


You are +1 for me being a maintainer or for a new project? I'm joking :)


Do we have some examples of stuff that could go there?


Sure, timing a request! :) More seriously a hook which would implement various retry strategies would be very good way to show power of hooks. I think we could find a couple more ideas in all those closed issues but I don't remember any right now.


Cool. I didn't have a good mental picture of what would go there, but now that I do, I'm +1 on this too.


@piotr-dobrogost both ;)

Also, if a separate repository is not acceptable to @kennethreitz, my next choice would be to add a sort of "Recipes" section to the docs which could include hook recipes.


And I just realized that with hooks, I could add a hook to my session object in github3.py to process the args and if `files` is not specified, use `json.dumps` on data. ... Now to ponder on whether that magic is a good idea or not... 


I'll have to think about this :)
__version__ should be set to `dev` unless tagged

Currently after `pip install git+git://github.com/kennethreitz/requests`, `pip freeze` shows `requests==<THE_NEWEST_RELEASED_VERSION>` which is misleading as this is true only until the first commit after release :) For this reason I think `__version__` should be `dev` in all untagged changesets. This is what urllib3 does for example. Or even better yet version could be `<THE_NEWEST_RELEASED_VERSION>.post1` or `<THE_NEXT_PLANNED_VERSION>.dev1` or similar.


For my personal projects I usually use `next-planned-release-number.b` for beta. This way I can push a beta version to PyPI for people to test and make it obvious that it is not guaranteed to be stable.


I disagree — Python Packaging doesn't support these types of versions officially, and it causes additional maintenance.


@kennethreitz [PEP386](http://www.python.org/dev/peps/pep-0386/) actually allows for the syntax I suggested. See this [section](http://www.python.org/dev/peps/pep-0386/#the-new-versioning-algorithm) in specific.


Interesting. I'm still not going to maintain that though :)


Ok.


> Python Packaging doesn't support these types of versions officially

True, `dev` is not supported. The other two were slightly off too - I corrected them.

> and it causes additional maintenance.

You mean one additional commit after tagging marking version as development one? I wonder if this could be automated using git hook attached to tagging operation?


No thank you.
Add parameter to iter_content to disable content decoding

I wanted to use requests to transfer a file chunk by chunk. However, before I transferred the body, I copied a header from the original response to the new response. Since request decodes gzip and deflate, the header was wrong and my content couldn't be interpreted by the browser.

This patch adds an option to `iter_content.` to disable decoding of gzip or deflate content.


`iter_content()` is supposed to always decode thus adding parameter changing this is not the right thing to do and I guess it won't be accepted. I thing there was some discussion about adding similar method for raw data but the idea was rejected. You could find this discussion in closed issues and voice your concern there. Isn't `raw` what you need?


Okay, thanks for the input. I have a solution for myself but thought that other people might be interested in this as well. I'll close the issue then.


What's your solution?


1st: Copy the code from `iter_content` but without the call that deflates or unzips.
2nd (hack): Set the `content-encoding` to `""` before calling `iter_content`.


Interesting. See #844 for a related issue (related to byte encodings of Unicode codepoints, rather than gzip).
Attach Content-Length to everything.

Closes #223

Sorry this took so long.


Merging as soon as the build passes :)


TypeError: object of type 'file' has no len()


Yeah, didn't expect there to be a failed test. Just ran them and this should pass. Sorry about that.


No worries! 


Bah, too exhausted to deal with this tonight, I'll need think about a better way to handle python 2 and 3 tomorrow. And I need to reinstall python 3 on my machine so I don't have to rely on Travis 


:cake:


:beers: 


@sigmavirus24 Re: handle Python 2 and 3, I'd like to suggest [Pythonbrew](https://github.com/utahta/pythonbrew).


@Lukasa very interesting. Thanks! I'll look more closely at it later tonight or tomorrow.
Specify SSL Protocol Version

I pretty sure this may fix some of the ssl errors regarding clients trying to connect etc.

I base this primarily on pythons ssl wrap functionality from 2.7 http://docs.python.org/2.7/library/ssl.html#ssl.wrap_socket

and openssl's CLI server test tools

openssl s_client -connect servername:443 -ssl3
openssl s_client -connect servername:443 -ssl2
...
...

Clearly this would be a reverse compat feature, but I'm hoping there are enough devs who crave the lower level ssl control  of ssl.wrap_socket with this libs tidy stability. 

Mmmmmm bound ssl sockets :+1:  [http://docs.python.org/2.7/library/ssl.html?highlight=ssl#client-side-operation]

Thanks for reading :)


Thanks for the suggestion!

Kenneth is working on refactoring requests to use Transport Adapters (see #895), which will allow specification of options like these. For this reason, we're not accepting any Pull Requests that add this functionality, since they'll cease to work when we make the move.

After the change, however, one of the most highly requested features is proper handling of SSL versions, including falling back to older versions (like browsers do), so we'll likely revisit this issue again.
Fixes so that safe_mode works for Sessions

This fixes issue #888. That safe_mode doesn't work when doing requests from a Session instance.

Could be fixed in a number of different ways of course, and if you don't like the way I've done it here, the failing test should still be useful.


Are you aware of issue #639 and my pull request #671 fixing this?


Piotr, why not reopen your PR then? 


Well I haven't received any response to https://github.com/kennethreitz/requests/pull/671#issuecomment-8787392 yet...


@piotr-dobrogost I saw your pull request after I had made the fix, and since it was closed I figured I could open this one. I don't really care which solution is merged as long as it solves the problem :).


I recall @kennethreitz first said he did not like the patch, then after a while he said he actually could accept it but then nothing more happened. If you compare our pull requests you notice that your change to `Session._send_request()` is not enough as this is not being called when `return_response` param is false (https://github.com/kennethreitz/requests/blob/v0.14.2/requests/sessions.py#L237) The change must be made on `Request.send()` level which always is called and that's what I did in my patch.


Thanks!


@kennethreitz 

I'm curious why didn't you merge my 6 months old pull request #671 which solves problem for all cases but you merged this one which does not? Thanks.


@piotr-dobrogost the new one is far simpler. The less code the better.


The new one is not correct to begin with. I guess this is more important than amount of code...
“Everything must be made as simple as possible. **But not simpler.**” ― Albert Einstein

> the new one is far simpler.

I hope you don't count adding 13 lines of code to **tests** instead 51 lines of code in my pull as being simpler...


What's wrong with this one?


It does not fix the problem when `return_response` is false. I explained this in my [comment](https://github.com/kennethreitz/requests/pull/953#issuecomment-10677611) above.


I don't see how this is incompatible?


Incompatible with what? Which part of my [comment](https://github.com/kennethreitz/requests/pull/953#issuecomment-10677611) is not clear?


The problem seems to be that if you make a request where `return_response` is set to `False`, and then subsequently call its `.send()` method, that method call can still throw exceptions. I don't know why anyone would actually do that instead of delaying their function call, but I guess they might.


This is how grequests works, which is why it was added :)

This API is being reworked for the refactor, however. Let's defer to that.
Streaming Requests

Currently refactoring the codebase for v1.0. Cleaning things up.

Plan:
- Replace the current `prefetch` parameter with `stream`. 
- Make all uploads stream.

Is there any downside to streaming all uploads? @mitsuhiko quick opinion?

---

Related: #895


If you're streaming files, I don't see a problem. If you're iterating over generators, then you don't know the content-length in advance and you either need to send the less-supported chunked encoding or ask for the length in advance.


I don't see anything objectionable about requiring a Content-Length in 
advance.


Nah, bytestrings would simply be uploaded as they currently are.

Generators would use chunked encoding, and potentially have an option for content-length (or they could simply set the header themselves and we'd recognize that)


I meant for streaming uploads ;)


Streaming uploads will happen soon, but not for 1.0.


Open again! 

Let's do this.


:metal:


Turns out we already support file-like objects. This is going to be super simple.


Done.
Eventlet support

First at all, thanks for porting requests to the asynchronous world, I loved it! 

However I'm using Eventlet instead of Gevent so I adapted the code to make
it work with Eventlet. You can see it at https://github.com/jmg/erequests.

I was wondering if we can make a unified version where the library can
detect or be configured to use either Eventlet or Gevent based on the user
needs. 

Many thanks.


Sorry, I wanted to open this on the grequests repository.
Only return a path if the cacert.pem file exists.

This will permit the deletion of just that one file in order
to fall back to the [probably more accurate but less consistent]
Distro provided CA certs.  

I need to be able to add our internal CA and I don't want to have to patch your module every time I upgrade.  Personally, I think it is backwards to prefer your own CA certs with no good way to prefer
the OS/Distro provided certs.  I realize it makes your testing much easier since you have a known
list of certs.  Providing an environment override is a poor substitute to an actual API.


If that file doesn't exist, it return from that function should be `None`, correct?

There is an actual API for providing your own CA bundle, it's just not well documented ;)


No, the return from that function is the filename, whether or not the file exists.  And I figured out I could pass my own CA bundle path in the verify parameter, but in my case, that would mean I'd have to patch the module that calls this code, so it seems easier to make "requests" DTRT instead.  ;-)


This seems hacky, but I'll accept it just for you.
Cookie error

After upgrading from 0.10.x to the newest, I'm getting the following error on some requests:

https://gist.github.com/6efe9228a53f0e3d8270

Note that this request uses oAuth add-on but I'm experiencing it without it as well. Python 2.7.2


Yup, that's pretty unlikely to be an OAuth issue.

I can't look into this while I'm at work, but I'll take a pass at it when I get home.


So I lied slightly.

Out of interest, can you try uninstalling and reinstalling Requests (`pip uninstall requests` followed by `pip install requests`). It looks like Requests had some major cookies related changes in V0.12, and the most pertinent commit seems to be 4d6871d9176c13affe625b1885278d396a39f21d. This should turn all `dicts` passed in as the cookie parameter into `CookieJars`. In your traceback this doesn't seem to have happened, so I just want to rule out the possibility that this code is absent in your install.


I started to doubt myself if I misread one previous error and tried this with buildin OAuth1 support instead of requests-oauth and it worked. requests-oauth hasn't been updated in 5 months so it's most likely out of date. Closing.
Respect the no_proxy environment variable.

This change is in response to issue #879.

The `no_proxy` environment variable is a pretty strange beast, so this code is a little less elegant than I'd like.

I also concluded that the right place to handle it was in `get_environ_proxies()`, but it might be better to hoist some of this code out and handle it in `models.py`. If you think that's a better idea, let me know and I'll make the changes to do that instead.


Wow, Travis has kind of imploded here.

I don't really know what to do about this, the tests all pass fine for me.


This is what I get for not running the tests using `nose`. =D All better now!


@Lukasa py.test is still the best :)


Nice, thanks!


+1 py.test is the best.
Setting timeout on celery worker causes TypeError when attempting to throw TimeoutError 

# Problem

Setting a request timeout causes a TypeError when attempting to throw TimeoutError  which is inherits from PoolError.
# Possible solution

Set a default message in the PoolError constructor.

```
#def __init__(self, pool, message):
def __init__(self, pool, message=''):
```
# Traceback

```
 [2012-11-17 15:47:24,980: ERROR/MainProcess] Unrecoverable error: TypeError('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.TimeoutError'>, ("HTTPConnectionPool(host='looney.com', port=80):   Request timed out. (timeout=1.0)",))

 Traceback (most recent call last):
   File "/home/user/virtpy/lib/python2.6/site-packages/celery/worker/__init__.py", line 347, in start
     component.start()
   File "/home/user/virtpy/lib/python2.6/site-packages/celery/worker/consumer.py", line 390, in start
     self.consume_messages()
   File "/home/user/virtpy/lib/python2.6/site-packages/celery/worker/consumer.py", line 474, in consume_messages
     readers[fileno](fileno, event)
   File "/home/user/virtpy/lib/python2.6/site-packages/billiard/pool.py", line 669, in handle_event
     self._it.next()
   File "/home/user/virtpy/lib/python2.6/site-packages/billiard/pool.py", line 642, in _process_result
     ready, task = poll(timeout)
   File "/home/user/virtpy/lib/python2.6/site-packages/billiard/pool.py", line 1059, in _poll_result
     return True, self._quick_get()
 TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.TimeoutError'>,("HTTPConnectionPool(host='looney.com', port=80): Request timed out. (timeout=1.0)",))
```


Oops, this appears to be a bug in urllib3 not requests...  I will attempt to place a pull request on that project.


Heh, fair enough. Thanks for the bug report anyway! =)
Tracking: 'requests' org

A GitHub org for Requests has been created.

http://github.com/requests

The purpose of this org is to keep track of the assets and supporting libraries around Requests.

Examples:
- requests-ntlm
- requests-oauth
- requests-docs-jp
- requests-docs-fr


As @kennethreitz pointed out, the org needs a logo. I'm design-blind and have no taste, so someone else should take that on. =D


I can't design anything either. What about the owl on docs.python-requests.org ?


I think that's a sea turtle.

On Thu, Nov 15, 2012 at 1:43 PM, Ian Cordasco notifications@github.comwrote:

> I can't design anything either. What about the owl on
> docs.python-requests.org ?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/938#issuecomment-10426849.


`\(o_O)/` I'm apparently blind. 


This will mostly be a documentation change. 

These changes can be made today.


What stuff are you looking to have in the docs?


the new auth stuff, mostly :)


Wouldn't be a good idea to put the requests repository inside requests organization? (and updating the website link)


Any advantage from doing that would be severely outweighed by breaking links all over the internet. =)


I also have no interest in moving requests's repo, regardless of any benefits. It's my project, it belongs in my github profile :)


See, when I say that it sounds sycophantic. =P


> See, when I say that it sounds sycophantic. =P

This x2


:camel: 


:monkey: I'm bad at this game


:chart_with_upwards_trend: :penguin:


:metal: :metal: :8ball: 


:dancer: :dancers: :slot_machine: :sleeping: 


:octocat: :octopus: :baby: :baby_bottle: :100: 


:accept: :baby_chick: :calling: :dash: :earth_africa: :fallen_leaf: 


I don't know how to make pretty pictures (sorry to break the chain), but I moved RestSharp out of my personal account into an org and I regretted it greatly (broken links, misdirected pull requests/issues, etc). So +1 to not moving kennethreitz/requests from where it is.
Fix for default_user_agent when SIGCHILD intercepted 

In environment, where SIGCHILD interrupted (e.g. running code under gunicorn) — platform.system() and platform.release() causes `IOError: [Errno 10] No child processes`

Tail of the stacktrace in this case:

```
File "/Users/cleg/Projects/nimble-all/server/eggs/braintree-2.14.1-py2.7.egg/braintree/util/http_strategy/requests_strategy.py", line 2, in <module>
    import requests
  File "/usr/local/lib/python2.7/site-packages/requests/__init__.py", line 53, in <module>
    from .models import Request, Response
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 27, in <module>
    from .defaults import SCHEMAS
  File "/usr/local/lib/python2.7/site-packages/requests/defaults.py", line 34, in <module>
    'User-Agent': default_user_agent(),
  File "/usr/local/lib/python2.7/site-packages/requests/utils.py", line 546, in default_user_agent
    '%s/%s' % (platform.system(), platform.release()),
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/platform.py", line 1303, in release
    return uname()[2]
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/platform.py", line 1250, in uname
    processor = _syscmd_uname('-p','')
  File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/platform.py", line 1007, in _syscmd_uname
    rc = f.close()
IOError: [Errno 10] No child processes
```

I've implementer simple workaround to make this code workable in any circumstances. Maybe it's not the best solution in this case, but it does the trick. Maybe better add general error handling to whole  `default_user_agent` function. 


Seems reasonable. Thanks!
Provide response.request.text to show body of a POST request

It would be helpful if there was a {response}.request.text or {response}.request.body function to show what was sent in the body of a POST request after url-encoding.

Great library, by the way.


This becomes tricky. 

If you look around in `models.py` you'll see where the body is constructed. It could very possibly be a file-like object. With that in mind loading it into memory twice can be quite expensive, especially if we store it ad infinitum. While I like the idea, I don't think it is exactly feasible. Sorry.

On the other hand, I can not close issues and have no "official" status inside the project, so mine is just an opinion. Feel free to continue to advocate for this. 


@sigmavirus24 is correct
! Fixes packaging and installation problems under Python 3.

Long short story, the associated commit aims at fixing #916 by refactoring the setup script and moving dependencies out of the package itself... These are now downloaded from PyPI whenever necessary before installation (via `install_requires`) while keeping compatibility with Python 2 and Python 3 in the same codebase (`chardet` is now resolved via two external requirements file).

The only downside of this is that one cannot import `requests` in `setup.py` (since that would require importing `charder`, which is not available at that time). Thus, the version is repeated both in `setup.py` and `requests/__init__.py`. I'll have a go at that once I find some more time, but for now this seems like acceptable for me.

**Note:** The package's version had been kept as `0.14.1` in the previous release. I decided to amend that to `0.14.2` instead of `0.14.3.dev0` or something like that, since maybe @kennethreitz wants to manage that himself. :-)


Not to be a downer, but I'm fairly certain @kennethreitz wants to keep everything vendored.


@sigmavirus24: In my opinion, the only scenario where dependencies should be kept vendored would be the case where they'd need to be modified from their original distributions in any way... For everything else there's `pip` with a requirements file or `setuptools`'s `install_requires`. Or am I missing something here?


Vendoring, like static linking, eliminates a whole class of bugs caused by slightly different behaviour in different versions of a library. As an additional advantage, it means that if there is a bug that appears to be caused by a dependency, the maintainer(s) know exactly what version of the dependency is being used, and they don't have to replicate the environment on someone else's machine. (Speaking from personal experience, this can be a nightmare, especially because 90% of bug reports do not list the versions of the dependency they have installed.)

There _might_ be an argument for not vendoring chardet, though. I don't have enough familiarity with why chardet was vendored to answer that.


Well put, @Lukasa. This bug was likely caused by a small accidental change to setup.py


@Lukasa: Hmm I see... If @kennethreitz then decides to keep the packages vendored I can try to tackle this problem by other means.


@bmcustodio that is the plan :)


@kennethreitz: Ok, so I'll have a look at that as soon as possible! :-)