move chardet call to the crawler