removeDups can now handle METHOD_FAIL and URL_FAILs with no matches correctly.