A more complete robots.txt parser