Handle marker removal while maintaining the marker invariant