changes to crawler
adding directories