Requiring headless.
Changing example url.
dsl common test spec
note that vanilla string compression does not yet work, only objects
bloomfilter turned off by accdient
debugging bloomfilter: still not working as expected in batch crawl and
mayb e not crawl job

needs to fix
will eventually try to replce redis bloomfilter with lua bloomfilter or
bloomd c-based server
added local saved_urls to cache until after post_batch / s3 save
a bit wasteful for memory if we are saving already to s3_cache
could use memory optimization here
debugging bloomfilter
Merge remote-tracking branch 'upstream/master' into new_headless
minor bug, page store was being saved even if save_batch is not
specified.  must specify save_batch to save the page store
Merge remote-tracking branch 'upstream/master' into new_headless
Adds headless option for page's js content.

- Option recrawls a page given proper http responses.
- rspec integration test must have the spec/headless_test_endpoint.rb up
  and running on port 80 for now.  Use rvmusdo ruby
headless_test_endpoint.rb for now.