Crawler is a straightforward gem that will index a given domain, and store the results in redis.