Handle corner case in lock being cleaned up and then removed again