meteor/meteor — pull request #411 by ayal (2012-10-20 18:54:29+00:00)
Title: serve facebookexternalhit user agent with spiderable

This change will make Spiderable serve facebookexternalhit user agent.

Using Spiderable after this change would enable one to manipulate the

tag in "client code", which would then actually be served to the Facebook bot when it comes looking for "open graph meta tags" (i.e https://developers.facebook.com/docs/opengraphprotocol/)

The main code change is adding "facebookexternalhit" to the condition under-which the phantomjs process is initiated. Other than that there are two other changes:

1. A rewrite of the code that re-assembles the request url without the "_escaped_fragment_" query param. This is needed because in the current version of Spiderable the code assumes there is always a query string (which might not be the case in the proposed version)

2. A small bug-fix to the regex test for html validity of the phantomjs response page (html openning tag might contain attributes, especially after client manipulation, which is the case here)