Delay construction of highest bit cache until it is actually needed.