Actually remove the layer and also make sure that the operation occurs on the layer management operation queue.