make some functions as independent functions from crawler class