updates to support easier integration of new datasets