Package: scrapy-manipulate-request
Scrapy Manipulate Request Downloader MiddlewareThis is an async scrapy request downloader middleware, support random request and response manipulation.With this, you can do any change to the reqeust and response in an easy way, you can send request by tls_client,pyhttpx, requests-go, etc. You can even manipulate chrome by selenium, undetected_chrome, playwright, etc. withoutany thinking of the async logic behind the scrapy.Installationpip3installscrapy-manipulate-requestUsageYou need to enableManipulateRequestDownloaderMiddlewareinDOWNLOADER_MIDDLEWARESfirst:DOWNLOADER_MIDDLEWARES={'scrapy_manipulate_request.downloadermiddlewares.ManipulateRequestDownloaderMiddleware':543,}Notice, this middleware is async, that means it is affected by some scrapy settings, such as:CONCURRENT_REQUESTS=16To manipulate request and response, it is very simple and convenient, just add manipulate_request functionin your spider, and send it to the meta, it's something like parse function.importscrapyclassTestSpider(scrapy.Spider):name="test"defstart_requests(self,):meta_data={'manipulate_request':self.manipulate_request}yieldscrapy.Request(url="https://tls.browserleaks.com/json",meta=meta_data)defmanipulate_request(self,request,spider):# return None, the requesst will be ignored# return scrapy.http.HtmlResponse or scrapy.http.TextResponse object,# the process of handle response will be started.passdefparse(self,response):passUseful ExampleSend request by tls_client in order to bypass ja3 verificationimportscrapyimporttls_clientfromscrapy.httpimportTextResponseclassTestSpider(scrapy.Spider):name="test"defstart_requests(self,):meta_data={'manipulate_request':self.manipulate_request}yieldscrapy.Request(url="https://tls.browserleaks.com/json",meta=meta_data)defmanipulate_request(self,request,spider):url=request.urlheaders=request.headers.to_unicode_dict()tls_session=tls_client.Session(client_identifier='chrome_112',random_tls_extension_order=True)proxy='http://username:password@ip:port'raw_response=tls_session.get(url=url,headers=headers,proxy=proxy)response=TextResponse(url=request.url,status=raw_response.status_code,headers=raw_response.headers,body=raw_response.text,request=request,encoding='utf-8')returnresponse# return None, the requesst will be ignored# return scrapy.http.HtmlResponse or scrapy.http.TextResponse object,# the process of handle response will be started.defparse(self,response):passMore and detailed tls_client usage seePython-Tls-Client.Use undetected chrom to operate webpageimportscrapyfrompprintimportpformatfromscrapy.httpimportHtmlResponsefromseleniumwireimportundetected_chromedriverasucclassTestSpider(scrapy.Spider):name="test"defstart_requests(self,):meta_data={'manipulate_request':self.manipulate_request}yieldscrapy.Request(url="https://tls.browserleaks.com/json",meta=meta_data)defmanipulate_request(self,request,spider):chrome_options=uc.ChromeOptions()chrome_options.add_experimental_option()chrome_options.add_argument()chrome_options.add_extension()seleniumwire_options={'proxy':{'http':'http://username:password@ip:port','https':'https://username:password@ip:port',}}browser=uc.Chrome(version_main=108,options=chrome_options,seleniumwire_options=seleniumwire_options,headless=True,enable_cdp_events=True)browser.set_page_load_timeout(10)browser.maximize_window()browser.add_cdp_listener('Network.requestWillBeSent',self.mylousyprintfunction)browser.execute_script()browser.execute_cdp_cmd()browser.request_interceptor=self.request_interceptorbrowser.get("https://tls.browserleaks.com/json")elements=browser.find_elements()...raw_response=browser.page_sourceresponse=HtmlResponse(url=request.url,status=200,body=raw_response,request=request,encoding='utf-8')returnresponse# return None, the requesst will be ignored# return scrapy.http.HtmlResponse or scrapy.http.TextResponse object,# the process of handle response will be started.defmylousyprintfunction(self,message):print(pformat(message))defrequest_interceptor(self,request):request.headers['New-Header']='Some Value'delrequest.headers['Referer']request.headers['Referer']='some_referer'More and detailed chrome operations seeundetected-chromedriverandselenium-wire.