changes to run over full dataset