[WIP] Make conftest remove tests instead of skipping.

Fixes #976.

Let's see if this workaround actually works. 
It *appears* to work from my testing. Might be a good idea for you to pull this branch and see if you hit any bumps.