remove defunct and uninteresting URLs