http adapter
A class that can handle a page
A simple storage engine based on mongo
A bloom filter based url tracker
A url tracker based on redis set
First raw working version
Removing test files
using standard gem files
Removing unused attr
Code cleanup
Code cleanup
File renamed
Dev Null storage added
Add support for compression
Wrapper for storage engine
wrapper for url_tracker
Home page var added
Fix compression method
Add support for gzziped page
Add my_test folder to the ignore list