improving duplicate filter, removing www. www1., etc; excluding editthis.info