Avoid potential race condition when remove is called twice