Disable the brick cache while we are precomputing minmaxes.

The precomputation shouldn't count as an "access".