Some changes made to crawler to handle unicode data.