Always reset the layer of things removed from storage

Fixes #6470