Only consume if we are empty.