Now extract feeds skips urls ending with robots.txt
fixed some bugs