Merge branch 'hotfix-0.7.2' into next
add expiration to Redis storage engin
adding patch from gnapse/anemone
add test for stop crawl
Update README.rdoc
Merge branch 'next' of github.com:efrat-safanov/anemone into next
add some memory limitation on queues
Merge pull request #1 from efrat-safanov/limit_queue_size

add some memory limitation on queues
improve redis performance
Merge pull request #2 from efrat-safanov/improve_performance

improve redis storage performance
changeing stop_crawl behaviour to stop new links, but finish processing all pages in queue
Merge pull request #3 from efrat-safanov/stop_crawl_bugfix

changeing stop_crawl behaviour to stop new links, but finish processing ...
deadlock fix - pages queue could be refilled by waiting threads
Merge pull request #4 from efrat-safanov/hotfix_pages_queue_stuck

deadlock fix - pages queue could be refilled by waiting threads
Clean keys that are not used anymore on exit

Sometimes it makes sense to clean up after ourselves. This prevents Redis database from getting too big, if we use different prefixes. For example, when doing concurent crawls from different processes, you might want to give them different prefixes so keys don't collide. If you base those prefixes on something like process id, thread id or timestamp, there's a good chance you will shoot yourself in the foot and your Redis will go tits up quite soon. This commit prevents that, when using Redis backend. Similar changes would be needed for other storages.
Merge branch 'master' of git://github.com/efrat-safanov/anemone into next
Merge branch 'next' of git://github.com/efrat-safanov/anemone into next

Conflicts:
	lib/anemone/storage/redis.rb