build index properly for on the fly datasets