Improved fault tolerance in robotsParser