Check crawlability of web sites