updates to crawler