Makes tests play nice

**Not ready for merge**
This is opened for visibility and assistance in testing

cc: @aaronschachter 

related to: #1746