run over 12 hour period and optimize that we don't arbitrarily generate
the idx file each time, when we need not to