Harden comment using greedy matching. Added more tests.

Added extra tests to test robots.txt with comment and with multiple UA.

Also removed p.py since it has been renamed to scanner.py