refactoring and including new features as throttle
Merge branch 'master' of github.com:ssola/crawly