Package: curl-cffi
curl_cffiPython binding forcurl-impersonateviacffi.Documentation|中文 README|Discuss on TelegramUnlike other pure python http clients likehttpxorrequests,curl_cffican
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.FeaturesSupports JA3/TLS and http2 fingerprints impersonation.Much faster than requests/httpx, on par with aiohttp/pycurl, seebenchmarks.Mimics requests API, no need to learn another one.Pre-compiled, so you don't have to compile on your machine.Supportsasynciowith proxy rotation on each request.Supports http 2.0, which requests does not.Supports websocket.libraryrequestsaiohttphttpxpycurlcurl_cffihttp2❌❌✅✅✅sync✅❌✅✅✅async❌✅✅❌✅websocket❌✅❌❌✅fingerprints❌❌❌❌✅speed🐇🐇🐇🐇🐇🐇🐇🐇Installpip install curl_cffi --upgradeThis should work on Linux, macOS and Windows out of the box.
If it does not work on you platform, you may need to compile and installcurl-impersonatefirst and set some environment variables likeLD_LIBRARY_PATH.To install beta releases:pip install curl_cffi --upgrade --preTo install unstable version from GitHub:git clone https://github.com/yifeikong/curl_cffi/
cd curl_cffi
make preprocess
pip install .UsageUse the latest impersonate versions, do NOT copychrome110here without changing.requests-likefromcurl_cffiimportrequests# Notice the impersonate parameterr=requests.get("https://tls.browserleaks.com/json",impersonate="chrome110")print(r.json())# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}# the js3n fingerprint should be the same as target browser# http/socks proxies are supportedproxies={"https":"http://localhost:3128"}r=requests.get("https://tls.browserleaks.com/json",impersonate="chrome110",proxies=proxies)proxies={"https":"socks://localhost:3128"}r=requests.get("https://tls.browserleaks.com/json",impersonate="chrome110",proxies=proxies)Sessions# sessions are supporteds=requests.Session()# httpbin is a http test websites.get("https://httpbin.org/cookies/set/foo/bar")print(s.cookies)# <Cookies[<Cookie foo=bar for httpbin.org />]>r=s.get("https://httpbin.org/cookies")print(r.json())# {'cookies': {'foo': 'bar'}}Supported impersonate versions, as supported by myforkofcurl-impersonate:However, only Chrome-like browsers are supported. Firefox support is tracked in#59.chrome99chrome100chrome101chrome104chrome107chrome110chrome116[1]chrome119[1]chrome120[1]chrome99_androidedge99edge101safari15_3[2]safari15_5[2]safari17_0[1]safari17_2_ios[1]Notes:Added in version0.6.0.fixed in version0.6.0, previous http2 fingerprints werenot correct.asynciofromcurl_cffi.requestsimportAsyncSessionasyncwithAsyncSession()ass:r=awaits.get("https://example.com")More concurrency:importasynciofromcurl_cffi.requestsimportAsyncSessionurls=["https://google.com/","https://facebook.com/","https://twitter.com/",]asyncwithAsyncSession()ass:tasks=[]forurlinurls:task=s.get(url)tasks.append(task)results=awaitasyncio.gather(*tasks)WebSocketsfromcurl_cffi.requestsimportSession,WebSocketdefon_message(ws:WebSocket,message):print(message)withSession()ass:ws=s.ws_connect("wss://api.gemini.com/v1/marketdata/BTCUSD",on_message=on_message,)ws.run_forever()curl-likeAlternatively, you can use the low-level curl-like API:fromcurl_cffiimportCurl,CurlOptfromioimportBytesIObuffer=BytesIO()c=Curl()c.setopt(CurlOpt.URL,b'https://tls.browserleaks.com/json')c.setopt(CurlOpt.WRITEDATA,buffer)c.impersonate("chrome110")c.perform()c.close()body=buffer.getvalue()print(body.decode())See thedocsfor more details.scrapyIf you are using scrapy, check out these middlewares:tieyongjie/scrapy-fingerprintjxlil/scrapy-impersonateAcknowledgementOriginally forked frommultippt/python_curl_cffi, which is under the MIT license.Headers/Cookies files are copied fromhttpx, which is under the BSD license.Asyncio support is inspired by Tornado's curl http client.The WebSocket API is inspired bywebsocket_client.[Sponsor] Bypass Cloudflare with APIYescaptcha is a proxy service that bypasses Cloudflare and uses the API interface to obtain verified cookies (e.g.cf_clearance). Clickhereto register:https://yescaptcha.com/i/stfnIO[Sponsor] ScrapeNinjaScrapeNinjais a web scraping API with two engines: fast, with high performance and TLS
fingerprint; and slower with a real browser under the hood.ScrapeNinja handles headless browsers, proxies, timeouts, retries, and helps with data
extraction, so you can just get the data in JSON. Rotating proxies are available out of
the box on all subscription plans.Sponsor