Protect the remove operation with lock.