included get_headers to verify if request is a file
crawler and download work together
remove unused files
download recursively
refactoring crawler
refactoring download.py