Changed to curl, added referrer and agent.
Optimized crawler function.
Added fix for websites with rates by letter urls.