[WIP] Make conftest remove tests instead of skipping.

Fixes #976.

Let's see if this workaround actually works. 
Also, this seems like it's a good motivator to switch to `@pytest.mark.requires_backend_interface(RSABackend)` or something?