fixed problem with link checking, added user-agent header

some servers respond with 403 messages to requests without
a user-agent header