put all urls together and then multl-threading, an ignore file is added