Always build the KDtree for fewer points