Clarity on docs regarding debugging

As spending about 10 minutes figuring out how to log raw request/responses in requests, I eventually came across the following [SO article](http://stackoverflow.com/questions/10588644/how-can-i-see-the-entire-http-request-thats-being-sent-by-my-python-application), which references an obscure part of the [documentation](http://docs.python-requests.org/en/master/api/?highlight=debuglevel).

It would be nice if there was a clear section in the docs which explains how to do this, as it's not very clear and took quite a bit of searching. A quick copy/paste (or a complete move) would suffice. Thoughts?


Using the search functionality with the phrase "logging" helped me find the section of the docs that details how to enable logging. I for one would rather not have the same content copied and pasted in the documentation because that provides two places where it would need to be updated and they will easily fall out of sync. My question then is, where do you think it would make most sense to move this to? 


Agreed on not duplicating instructions. I'm thinking a section in the "quick start" at the bottom, perhaps the title "Debugging" followed by a collection of all the tricks that can be used to enable the various levels of debugging. For other debugging recipes outside of basic logging usage, we could link to another page with the various recipes. Thoughts?


I really don't think this belongs in the quickstart section frankly. It isn't something most people will need as soon as they start using requests. It's more of an advanced usage piece of information, but even then our Advanced section is already quite large and overloaded. I'm not entirely convinced this shouldn't be it's own section though. I'd love to see more of the recipes you think belong there before we do that though and I'd would **really** like @Lukasa's input when they have a minute or two to spare.


Yeah ideally it just needs to be somewhere that can be easily found, because currently it's in an obscure place which is difficult to find. Perhaps "troubleshooting" or something like that... I'd argue that debugging requests/urllib3 is quite a tedious task, for example the logging approach dumps all the info but in a non human readable format... Recipes would be for showing how to dump info in a pretty format, for example request/response headers split with their new lines intact etc.


I think the bigger problem here is that the SO answer isn't correct. Requests has _no_ way to dump the request exactly as it was sent on the wire. There are a number of layers in between the top and us, any of which are free to change the code as they please. `httplib` is the worst offender here, particularly with responses, which it parses before dumping them (so if they don't parse properly they won't get dumped properly either).

I think a 'how to debug HTTP' article is a good idea, but it's not entirely clear to me that it belongs in our documentation. If we had it, however, I'd be recommending things like Wireshark/tcpdump, rather than logging. Don't trust what the application tells you it's sending, ask the OS instead.


@Lukasa All good points. From the perspective of a new user who isn't aware of the architecture of requests, they just want a simple mechanism that will allow easy debugging without having to use wireshark filters, passive burp proxy etc. For example, when using `curl` you can use the `-v` option to dump all request data, something like that for requests would be more than sufficient. 

I'm wondering if it would be more appropriate to raise a feature request in `urllib3` for a simple session method, such as `verbose(True/False)` which then implements the necessary recipe to dump traffic data in a similar format as `curl -v`. If accepted, then we could implement a method on `requests.Session` that called the urllib3 verbose method.

Does that sound like a good way forward?


No it doesn't sound like a good way forward. urllib3 doesn't need that parameter and we won't be expanding the Session API unnecessarily. I'm also rather perturbed that people find this so hard to find. If you find requests' docs and you search for "logging" you'll find that documentation quickly. It's not in the higher traffic parts of the docs, granted, but it's also not something I see a lot of people using very much.


Right, I'm going to disagree with you there, and I'm also somewhat "perturbed" that you feel this would be unnecessary expansion. If you have already acknowledged that people are finding it difficult, then perhaps that would suggest something. Furthermore, the logging shown from `httplib` is also not very human readable and isn't easily useful (imho).

I'm going to raise this in `urllib3` anyway, and see if the feedback is any different. 

Thanks for the replies all!


Cory's a maintainer there and I'm opposed to adding this to either our 
Sesssion API or urllib3's PoolManager API, so you can create the issue but it 
isn't likely to progress very far.


I'm fully aware of that, thanks.


@sigmavirus24 Normally I don't bother commenting on rudeness, but I felt compelled to on this occasion because of my prior background with requests/urllib3. I really feel like your response was overly aggressive and borderline rude. I've never had this issue with any of the other core maintainers in the past, and I spent time specifically replying because they have always been polite and reasonable, even when rejecting my code or requests. This interaction has left me feeling like I no longer want to bother contributing towards either of those projects, your reaction/approach was completely unnecessary and similar in tone with the Django core developers poor bed side manner.

Sometimes things get rejected, and that's fine, but there's no need to be rude about it.


Closing for now. For anyone in future, there has also been some discussion on this in https://github.com/shazow/urllib3/issues/107


@foxx I'm sorry you felt that way. It's never our intention to create a hostile atmosphere, and I apologise unreservedly if that's what we did.

This is another of my 'wish list' features for a rewrite of `httplib`. If I ever find the time to do it, this will be a key tentpole of that rewrite.


@Lukasa Thank you for the follow up, it's appreciated. Based on comments in those other issues and a quick inspection of the source, additional debugging support would _probably_ have to rely on monkey patching. Given that's not a great approach, I'd agree that a rewrite of `httplib` is sounding like the cleaner option. 

Out of curiosity, were you planning on keeping the `httplib` rewrite backwards compatible, or would it be `httplib2`?


@foxx I wasn't trying to be rude just honest. If there's anything that annoys me most it's when people aren't fully honest in a way that would help me prevent wasting my time. If I'm mistaken about which comment offended you, let me know. It's no excuse but I'm typically in a rush and replying from my phone. I don't often have the luxury of re-reading things to make sure they won't come across as rude. I do hope you continue to contribute because your input is valuable. 


@sigmavirus24 I appreciate the follow up, thank you, and no hard feelings :)


@foxx It would not be backwards compatible, but neither would it be `httplib2` (that already exists!). `httplib`'s API is _bad_, and it limits what can be done by libraries that build on top of it. I have ideas for where to go next, but again, no time. =(