Only decrement size if it is known.