Add some checking to prevent bad things on removals