Crawler handles non-html content