Revert [bb07fc6f1f408356], it causes crashes (bisected it, the result was this commit! No idea why! Help!). 
With this revert all test-cases pass again, but I don't know the code good enough to judge what's really going on here.