Only add to the queue when it is low