scan for canonical urls and shortlinks