various updates after testing with duncan's dataset