added the python crawler and deduplication function