you need to verify. Well, with different highlighting, this becomes is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. -- Dennis May 16 at 2:51 4 one could argue that even hitting robots.txt for our context is a bad way to act ... especially if we only need to point and shoot at a "single" link -- waffles May 16 at 2:51 @waffles: If you hadn't mentioned it, I wouldn't have even thought of checking robots.txt. -- Robert Harvey ? May 16 at 2:53 @Dennis: Relative to the total traffic on Facebook? It's infinitesimally small. -- Robert Harvey ? May 16 at 2:58 @TimManishEarth: Nevermind. I misunderstood parts of the question. Unless there is a big amount of links on a single website that have to be verified, I agree that robots.txt shouldn't be consulted. -- Dennis May 16 at 2:58 @Dennis: And even robots need to spend time on FB, don't they? ;-) -- Manishearth May 16 at 3:00 1 @TimManishEarth: I don't know. I never was a robot nor have I ever spent time on Facebook (sorted by likelihood to happen, decreasing). -- Dennis May 16 at 3:02 @waffles I don't see anything wrong with just hitting a link to verify it exists, and I see no reason to consult robots.txt prior to doing it, even if you have a large amount of links to check on a single domain. Just stagger