web crawler updated and changed directory structure