[WIP] Make conftest remove tests instead of skipping.

Fixes #976.

Let's see if this workaround actually works. 
Rather than doing a copy and remove, can we write this as "new empty list + append"?