Better testing for chunked uploads.

We have a whole code branch that does chunked uploads that is not and has never been tested. That's a problem, because sometimes it breaks and we don't notice (#2861). I'd like to add more testing for chunked uploads.

However, we can't use httpbin to do this testing. This is because the WSGI spec staunchly refuses to let applications see chunked transfer encoding at the app layer. For this reason, to test chunked transfer encoding requires a new style of testing, one that doesn't involve running a WSGI server but instead running something that can see the bytes _as they hit the wire_. Basically, something like the socket level tests of urllib3.


One possibility would be adapting urllib3's solution to our case. This new server we'd make doesn't need to support the whole HTTP protocol, because we just need it for the chunked transfer encoding thing, right? 


I think we wouldn't actually use a proper server at all, just a pair of sockets.


Right, ok. 

Both #2860 and this issue come from the same problem: pytest-httpbin's server doesn't seem to support the whole spectrum of our testing needs properly (not saying it is bad software, it's great). We could solve both issues at the same time just by making our socket server able to listen to listen to _parallel_ requests. The problem is that it would be necessary to hardcode the response headers in the threaded's auth case, which is far from optimal.


Even that is overdesigning it. All we need is to be able to receive a request and send a response. Parallel requests aren't a concern, just one is totally fine. We need a listening socket that can send a pre-canned 200 response when a request is received then read all the data. That's literally all we need to do.


Yes, I understand. My point was that we could solve #2860 too by making this server able to listen to parallel requests. It's ok though if you think it would be better to address those two problems separately.


Oh, I see. Frankly, I think we need to write a better test for #2860, because the approach we had been using (brute force running a lot of threads) is not really the most effective way to test the function.


Ok!

Would it be ok if I start a PR trying to adapt `SocketDummyServer` and `SocketDummyServerTestCase` from urllib3? 


You are welcome to, though it'll probably require quite a bit of work.


May it make sense to get this into urllib3 and also get rid of tornado there?


@ml31415 Given that we're going to try to move the chunked encoding logic to urllib3, that's probably true. However, the time investment here is still useful, because this kind of focused socket-based testing is better than using httpbin. =)


Agreed, that it makes sense to have this kind of testing in requests. But the testserver itself may be interesting for urllib3 too. So maybe put the testserver there, and requests just reuses it for testing? Instead of duplicating code around.


urllib3 already has its own equivalent of this. =) That's where we stole the idea from. However, it's a little bit nicer to avoid importing urllib3's test framework, because generally speaking testing functionality is not covered by versioning constraints, and can change at any time. Having that change under our feet without warning would be...annoying.


If I didn't overlook anything, urllib3 uses tornado for a bunch of testing (besides the dummyserver). This is quite some overhead imho, and could be done with a much simpler - own - test server implementation, e.g. extending the dummyserver, or adding some features here.


@ml31415 If urllib3 switches away from Tornado we'd like to switch to a socket-level test approach as you saw with the chunked encoding tests. That's something we're interested in doing, but it's hard to find the time to do the migration. If you're volunteering, then go for it!


Hmm, just had a closer look on it. Moving a bunch of the dummyserver tests to the socketserver (e.g. the retry tests), and replacing tornado with some pimped SimpleHTTPServer for the remaining tests seems quite possible... anyways, I guess that's another discussion then.