fixed some bugs with limiting crawling to domain only