finished a crawler using agents