Crawler unicode improvements, cleanDuplicates only searches complete file when there were missing parents