Package: scrapy-auto-translation-middleware
scrapy-auto-translation-middlewareOverview: why you need a middleware to do the translation workIn many circumstances you may want to automatically translate an Item field into another language, and write it into another field. For example, you have defined an Item:class City(scrapy.Item):
    name = scrapy.Field()
    total_area = scrapy.Field()
    per_capita_gdp = scrapy.Field()
    time_zone = scrapy.Field()The meanings are pretty straight forward. Let's assume you want the name field to be translated into some other languages, say, French, Simplfied Chinese, Japanese, by sending translation requests to Google Translation service. You obviously have an option to do this:Option 1:  do the translation in the spider (the worst approach)For example, in your spider:from some.google.translation.lib import Translator
Class CitySpider(scrapy.Spider):
    name = "cities"
    start_url = ["http://some.citydata.website",]
    def parse(self, response, **kwargs):
	    name = response.xpath("path.to.city.name").get()
	    translator=Translator()
	    name_zh=translator.translate(name, source="en", dest="zh-CN")This is a bad idea as you are making a synchronous request to Google when you are working in an asynchronous framework (in the case of Scrapy for specific, twisted).  A number of bad things will be resulted:The failure of Google Translation work will stop the entire crawling process, thus your spider will be much more vulnerable to unexpected events.As you are doing a synchronous work that may take unpredictable length of time, the spider will suffer from low performance (keep in mind that there's only one thread running in the Twisted framework).The downloader will not take care of the translation work so the states data will become inaccurate.Option 2: send a dedicated request to Google to finish the translation (much better, but tedious)Consider the following:import scrapy
class CitySpider(scrapy.Spider):
    name = "cities"
    start_url = ["http://some.citydata.website",]
    def parse(self, response, **kwargs):
	    name = response.xpath("path.to.city.name").get()
	    yield scrapy.Request(url="google.translation.url", cb_kwargs={"name":name})
	def translate(self, response, **kwargs)
	    name = kwargs["name"]
	    name_zh=response.xpath("path.to.name_zh").get()
	    yield items.CityItem(name=name, name_zh=name_zh)This a apparently much more in tune with Scrapy's design rules but it would be tiresome to introduce extra callback functions just for doing the  translation. Moreover, it would lower down the maintainability and readability of the code.Option 3: use a translation middleware (best one)By making use of a translation middleware you are allowed to do this in the item definition:import scrapy
class CityItem(scrapy.Item):
    name = scrapy.Field()
    name_zh = scrapy.Field(auto_translate=True, source="name", language="zh")
    name_fr = scrapy.Field(auto_translate=True, source="name", language="fr") 
    name_ja = scrapy.Field(auto_translate=True, source="name", language="ja")In your spider, you just need to populate the original information (the field "name") and the middleware will automatically handle others that are marked asauto_translate=True:......
def parse(self, response, **kwargs):
    ......
    yield items.CityItem(name="city_name_crawled_from_web")installationTo install scrapy-auto-translation-middleware, run:pip install scrapy-auto-translation-middlewareSettingsScrapy-auto-translation-middleware provides built-in support for Google Translation. If you are happy with it, add the following code into your project's settings.py file:SPIDER_MIDDLEWARES = {
    scrapy_auto_trans.spidermiddlewares.autotrans.GoogleAutoTranslationMiddleware': 701
}Google Translation requires an API Key for each translation service request so you need to specify the key in the settings.py as well:GOOGLE_CLOUD_API_KEY="<api.key.you.got.from.google.cloud>"If you don't feel comfortable to hard-code your API key in settings.py, another option is to specify the key as a command line option when you run the spider:scrapy crawl cities -s GOOGLE_CLOUD_API_KEY="<api.key.you.got.from.google.cloud>"Class hierarchyUsageThe most intuitive way of using the translation middleware is to subclassAutoTranslationMiddlewareBaseand re-write a number of methods, and specify the middleware onsettings.pyfile:class MyTranslationMiddleware(AutoTranslationMiddlewareBase):
    def translate(self, field_name, item, **kwargs):
	    """
	    This method is called for each field with auto_translate=True.
	    Returns either the translate result, or a (request, callback) tuple.
	    """
    def get_translate_result(self, response, field_name, item, **kwargs):
	    """
	    This method is responsible to interpret the response from the translation service website (e.g. Google Translate).
	    Returns either the translate result, or a (request, callback) tuple.
	    """Add the following in the filesettings.py:SPIDER_MIDDLEWARES = {
    scrapy_auto_trans.spidermiddlewares.autotrans.MyTranslationMiddleware': 701
}Define your itemsscrapy-auto-translation-middleware will be triggered if you setauto_translate=Truein the item field definition:class CityItem(scrapy.Item):
    name = scrapy.Field()
    name_zh = scrapy.Field(auto_translate=True, source='name', language='zh')You must specifysourceandlanguagein the field definition. They are pretty straight forward in their meanings.
The following options are supported:auto_translateThis is basically a tag for fields that should be translated by the middleware. Setauto_translate=Trueif you want the fields to be handled by the translator.on_failureUse this option to specify what you want to happen if a translation fails. The default action is REPORT_IN_FIELD.RAISE: raise the failure and let the exception handler to handle it.DROP_ITEM: silently drop the item.REPORT_IN_FIELD: write an error message into the target field,"--- translation error ---"by default.COPY_SOURCE: copy the source into the target field.SET_NULL: set the target field to be None (null in json).SET_EMPTY: set the target field to be an empty string.Example:from scrapy_auto_trans import FailureAction
class CityItem(scrapy.Item):
    name = scrapy.Field()
    name_zh = scrapy.Field(
        auto_translate=True, 
        on_failure=FailureAction.DROP_ITEM, # drop the item on translation failure,
        language='zh', 
        source="name",
    )translateUse this option to specify atranslator function. If specified, it will override the middleware'stranslate()method.
The translator function will be called when an item is being translated for a field. For example:def sqkm_to_sqmi(field_name, item, **kwargs):
    “unit conversion from sqkm to sq miles. this is a very quick translation"
    source = kwargs["source"]
    return float(item[source]) * 0.3861

def language_translator(field_name, item, **kwargs):
    "this will send a translation request to some website so we provide a callback to avoid taking too long time here"
	def callback(response, _field_name, _item, **cb_kwargs):
		return response.xpath("xpath.to.translation.result").get()
	source = kwargs["source"]
	return scrapy.Request(url="http://some.translation.service/?text=%s"%item[source]), callback

class CityItem(scrapy.Item):
	name = scrapy.Field()
	name_zh = scrapy.Field(auto_translate=True, translat=language_translator, language="zh")
	total_area = scrapy.Field()
	total_area_in_sq_miles = scrapy.Field(auto_translate=True, translate=sqkm_to_sqmi, source="total_area")In cases wheretranslateis present, all other field options (e.g. "source") will be provided to the translator function (and the callback function, if returned) inkwargs.languageUse this option to specify to what language the field should be translated.  If you are using Google Translate, the supported languages and the corresponding ISO language code are listed belowLanguage nameLanguage codeAfrikaansafAfrikaansafAlbaniansqAmharicamArabicarArmenianhyAzerbaijaniazBasqueeuBelarusianbeBengalibnBosnianbsBulgarianbgCatalancaCebuanoceb (ISO-639-2)Chinese (Simplified)zh-CN or zh (BCP-47)Chinese (Traditional)zh-TW (BCP-47)CorsicancoCroatianhrCzechcsDanishdaDutchnlEnglishenEsperantoeoEstonianetFinnishfiFrenchfrFrisianfyGalicianglGeorgiankaGermandeGreekelGujaratiguHaitian CreolehtHausahaHawaiianhaw (ISO-639-2)Hebrewhe or iwHindihiHmonghmn (ISO-639-2)HungarianhuIcelandicisIgboigIndonesianidIrishgaItalianitJapanesejaJavanesejvKannadaknKazakhkkKhmerkmKinyarwandarwKoreankoKurdishkuKyrgyzkyLaoloLatinlaLatvianlvLithuanianltLuxembourgishlbMacedonianmkMalagasymgMalaymsMalayalammlMaltesemtMaorimiMarathimrMongolianmnMyanmar (Burmese)myNepalineNorwegiannoNyanja (Chichewa)nyOdia (Oriya)orPashtopsPersianfaPolishplPortuguese (Portugal, Brazil)ptPunjabipaRomanianroRussianruSamoansmScots GaelicgdSerbiansrSesothostShonasnSindhisdSinhala (Sinhalese)siSlovakskSlovenianslSomalisoSpanishesSundanesesuSwahiliswSwedishsvTagalog (Filipino)tlTajiktgTamiltaTatarttTeluguteThaithTurkishtrTurkmentkUkrainianukUrduurUyghurugUzbekuzVietnameseviWelshcyXhosaxhYiddishyiYorubayoZuluzulanguageis optional and only necessary when your translator requires it. In cases where your translator is doing things like unit-conversion, for example, it would not be needed.
Usually you need to specify the language for both the source and target field. The target language must be explicitly specified. The middleware use the following order to determin the source language:languageoption in the source fieldThe middleware'sDEFAULT_LANGUAGEattributeDEFAULT_LANGUAGEin the project's settingsAmerican English ("en")sourceThis could be used to specified the field from which the translation is conducted.sourceis optional and only necessary when your translator requires it. In cases where your translator is doing things like unit-conversion, for example, it would not be needed.