1.fix proxy problem
2.handle robots.txt