Fix squlalchemy abstraction leakage

Sometimes a model is not a model and you can't set a model to None to
wipe it out, you have to set the relationship column id to None.

Also pin factoryboy so the tests run again,