on_page_error method added
make test to run faster
on_page_error block call
some code cleanup
some test added
removing obsolete comment
code cleanup
expose error trough page hash
run on_before_save right before storing the page
shorthand for HTTPSuccess response
some spec added
Merge pull request #29 from taganaka/on_page_error

On page error
Add example for error handling. #15
Fix variable name
Merge pull request #31 from tmaier/patch-1

Add example for error handling. #15 [ci skip]
robotex gem included
robots.txt support added
code cleanup
sample added
Merge pull request #30 from taganaka/robotstxt

Support for robots.txt