add a bunch of prototyping code
add a bunch of temp code
add a HTTP client for doing get requests, updated play to 2.1.1 / scala to 2.10.1, and got rid of some unused code
moved lots of Crawler code into an actor / Concurrent.unicast, updated HttpCrawlerClient with several more pipelines, refactored a bunch of junk
add a bunch of documentation to HttpCrawlerClient, and implement throttling on a per client basis, derived from the crawl-delay in the domain's corresponding robots.txt file
fix broken spray config fetcher
fix small bug
get basic robots.txt parsing working, add a hook to set custom port for testing
add some akka debug stuff
add a small standalone spec that uses a local http server for testing
wrapped a bunch of code in Try's in case of urls not being allowed for crawling
fixed several bugs (and refactored many methods to be wrapped in Try's) thanks to specs
small refactorings, more specs for http crawler client
small refactorings to add controller code for streaming crawler responses
buttload of refactoring, and a bit of extra documentation added
fixed a bug in streaming nulled out error json, added specs for streams trait, modified exception to include fromUrl, fixed some incorrect eofAndEnd default values
preliminary code to support manual redirect following + error handling
saving state
it compiles.
fixed a bunch of bugs, and (hopefully) fixed all NPEs in crawler client (ugh lazy vals)