Now extract feeds skips urls ending with robots.txt