[WIP] Make conftest remove tests instead of skipping.

Fixes #976.

Let's see if this workaround actually works. 
This works? I'm surprised skipping inside the generation section resolves this issue...