Avoid layer removal race conditions