1. add robots.txt parser and filter
2. sovled prob: robot policy filter makes the crawler VERY slow.
3. add file type filter