If you want to remove the duplicates in not such a big txt file(less than 1 million lines), recommend you to use removing2.py.