robots: avoid using a regexp for short user agents.
Miscellaneous optimizations.

This avoids using low-performance comamnds such @rx where possible.