modified logging and test script for the full dataset