Port test suite to pytest-httpbin

https://github.com/kevin1024/pytest-httpbin


Just attempted, but can't figure out how to use default funcargs with class-style tests.


 Duplicate of https://github.com/kennethreitz/requests/issues/2093 ?


Yup.


@kennethreitz yeah, pytest makes this a bit awkward, so I just released a new version of pytest-httpbin to make this easier.  Check out this section of the readme: https://github.com/kevin1024/pytest-httpbin#using-pytest-httpbin-with-unittest-style-test-cases

Here is the example usage:

``` python
import pytest_httpbin

@pytest_httpbin.use_class_based_httpbin
@pytest_httpbin.use_class_based_httpbin_secure
class TestClassBassedTests(unittest.TestCase):
    def test_http(self):
        assert requests.get(self.httpbin.url + '/get').response

    def test_http_secure(self):
        assert requests.get(self.httpbin_secure.url + '/get').response
```


That's perfect, thanks! This will allow us to just use it for the baseclass. 


Cool! Let me know if you run up against anything else.


Closing due to inactivity 


Aww too bad.  Was there something blocking this?  Would you merge a pull request adding this?


I'm still taking a look at this. Right now it's _mostly_ working, but there are a few tests that don't really function appropriately. Going to see if I can get the whole suite functioning again.


Ok, I don't think this will work just yet. There are a few problems in pytest-httpbin that I can't work around yet. The most severe one is that it seems to be unable to send chunk-encoded responses, which breaks a couple of our streaming tests. I consider this a bug in pytest-httpbin.

Also, `test_DIGESTAUTH_THREADED` has a tendency to fail miserably (usually with `EPIPE`). I suspect this is because pytest-httpbin is single-threaded. I'm not sure what to do about this yet, but we don't need to decide until everything else is working.


This also reveals a bug in the way we mock out `get_netrc_auth` in `test_basicauth_with_netrc`, which is unfortunate. We can work around that too.


Ok, bugs raised against upstream projects: kevin1024/pytest-httpbin#27 and Runscope/httpbin#259. Until this gets resolved and a new release of httpbin is pushed, however, I don't believe we can move forward with this. I've pushed my work to the pytest-httpbin branch of this repository for safe-keeping.


OK, the bug is fixed upstream in httpbin and a new version is released, 0.4.0.

Let me know how it goes with the threaded test.


Ok, so the threaded test is the only problem at this point: all the rest is green. I've pushed up the new pin and a minor fix to the pytest-httpbin branch.

At this point, I may need to play around with putting gunicorn or something similar in front of pytest-httpbin. It may be that something like Twisted or Tornado is a better choice here: anything that can meaningfully handle some number of concurrent connections.


Twisted can be used to run any wsgi app as I understand it. It may be worth attempting to run httpbin on top of that in that case.


Hmm, maybe I can use werkzeug's http server instead of wsgiref, since it's already a requirement of httpbin (-> flask -> werkzeug).


OK, I just released a multithreaded version of pytest-httpbin! The requests test suite seems to be running well for me with my updated version.

One thing to note though:  2 of the tests request a /delay/10 endpoint, and then time out before the response comes back.  You will see the error message from the HTTP server saying "broken pipe".  This is because the httpbin server tries to return the response to a client that has gone away.  Not really anything to worry about but might be scary for someone new to the test suite.


Alrighty, so I removed the threaded digest auth test entirely because it just doesn't behave well in this kind of environment. At some point we do need to go back and retest it, but I don't think we can do that until we totally revamp our testing.

In the meantime, the proposed switch to move offline is in #2859. We need to be really careful about when this gets merged to make sure there's no explicit calls to httpbin.org again, but we can do that.


I'm curious about the problems you were having with the threaded test.  Can you elaborate?


Two problems.

Firstly, the test had a tendency to vomit a lot of logs from the httpbin process into stdout, which ruins the test output. Not good.

Next, it failed a lot. About 50% of the time in my viewing.


@lukasa did you try the version I released a a couple weeks with a threaded server?  You shouldn't be getting test failures.  The error messages are still a problem though.  Noticed the requirements.txt in requests still has pytest-httpbin pinned to an older version so I'm wondering if you didn't see my above message from 14 days ago.  Regardless, I'm super excited you're using my library.  Cheers!


Thanks for the work @kevin1024! I did see the message, but the scary test output was a bigger problem. ;) I think it'd be better to come up with a different way to test this scenario.

Thanks for your library!


Oh well, it was worth a shot!


To be clear, @kennethreitz, the current test suite has been ported to use pytest-httpbin. =D At PyCon 2016 we'll probably want to have a discussion about the longer-term approach to testing in requests, but for now this is good enough. =)