Initial commit. Setup Go environment
Add gitignore. Add goconvey testing framework to env
First draft of simple data model for crawler
Add coverage tool
Ignore code.google.com directories
Minor refactor to page to improve testability. Add tests for dumping of site map.
Tweaked install of go-cover. Added goquery package
Initial commit of page processing and tests
Add processing for Javascript assets. Add protection from infinite loops both when referring to the local page and when via sub-pages
Complete work in ProcessPage. Added command-line to tool to pull together the pieces
Remove debugging prints. Added profiling code.
Add output with the number of CPU that will be used.
A mutex to page that is used when writing. doProcessPage now launches a gorouting per page
Check for variations of terminating slashes for the already visited pages. Only process html pages
Site to crawl is now a command line argument
Merge branch 'master' of github.com:yoink00/crawler