added parser options to robots.txt feature

as discussed in #95