few fix.

- added robots.txt
- removed login form link
- changes crawler library (cant run at perl 5.18 NOT RIGHT fix)