a web crawler that grabs links, selects them, and checks for exceptions