removes duplicates during processing, catch-all exception handles page load failures