Exploratory robots.txt crawler written in Go