Make the link finding/enqueuing process concurrent

Also add a timeout and handle content-type more intelligently