Package: scrapinghelper
ScrapingHelperThis library intends to make parsing HTML with using request_html.
So, When using this library you automatically get (same as request_html):Full JavaScript support!CSS Selectors(a.k.a jQuery-style, thanks to PyQuery).XPath Selectors, for the faint of heart.Mocked user-agent (like a real web browser).Automatic following of redirects.Connection– pooling and cookie persistence.The Requests experience you know and love, with magical parsing abilities.Async SupportAnd new features.URL / PROXY parser with validator.UserAgent manager using collected real user-agents.proxy rotation suppport.access with random wait timeshell environment options.logging support.Tutorial and Usageclass URLvalidator()unquote() / decode()enquote() / encode()In[1]:fromscrapinghelperimportURLIn[2]:u=URL()In[3]:u.validator('http://sample.om')Out[3]:TrueIn[4]:u.validator('http://sample.')Out[4]:FalseIn[5]:url=URL('http://www.example.com/sample?src=git&encode=jp')In[6]:url.is_validOut[6]:TrueIn[7]:url.attrsOut[7]:{'url':'http://www.example.com/sample?src=git&encode=jp','is_valid':True,'scheme':'http','netloc':'www.example.com','username':None,'password':None,'hostname':'www.example.com','port':None,'path':'/sample','params':'','query':'src=git&encode=jp','fragment':'','basename':'sample'}In[8]:url=URL('http://www.example.com/データ.txt')In[9]:url.attrsOut[9]:{'url':'http://www.example.com/%E3%83%87%E3%83%BC%E3%82%BF.txt','is_valid':True,'scheme':'http','netloc':'www.example.com','username':None,'password':None,'hostname':'www.example.com','port':None,'path':'/%E3%83%87%E3%83%BC%E3%82%BF.txt','params':'','query':'','fragment':'','basename':'データ.txt'}In[10]:url.queryOut[10]:'src=git&encode=jp'In[11]:url.get_query_val('src')Out[11]:'git'In[12]:url.set_query_val('src','csv')Out[12]:'http://www.example.com/sample?src=csv&encode=jp'In[13]:urlOut[13]:http://www.example.com/sample?src=git&encode=jpIn[14]:url.set_query_val('src','csv',update=True)Out[14]:'http://www.example.com/sample?src=csv&encode=jp'In[15]:urlOut[15]:http://www.example.com/sample?src=csv&encode=jpIn[16]:url.get_root_address()Out[16]:'http://www.example.com'In[17]:url.strip_query()Out[17]:'http://www.example.com/sample'In[18]:url=URL('https://ja.wikipedia.org/wiki/日本語')In[19]:urlOut[19]:https://ja.wikipedia.org/wiki/%E6%97%A5%E6%9C%AC%E8%AA%9EIn[20]:url.unquote()Out[20]:'https://ja.wikipedia.org/wiki/日本語'In[21]:url.decode()Out[21]:'https://ja.wikipedia.org/wiki/日本語'class UserAgentload_datafile(keep_user_agents: int=50, datapath: Optional[str]=None)get_random_user_agent()get_next_user_agent()attributesfirst_user_agentclass Scraperget_random_user_agent()  pass to UserAgent.get_random_user_agent()get_random_ipv4()get_random_ipv6()request()request_async()get_filename()get_links()get_texts()download_file()n[1]:fromscrapinghelperimportScraperIn[2]:sc=Scraper()In[3]:sc.get_random_user_agent()Out[3]:'Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.0 Safari/537.36'In[4]:sc.get_random_user_agent()Out[4]:'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/'In[5]:sc.get_random_user_agent()Out[5]:'Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.0 Safari/537.36'In[6]:sc.get_random_user_agent()Out[6]:'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1'In[7]:sc.get_random_user_agent()Out[7]:'Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.0 Safari/537.36'In[8]:sc.get_random_user_agent()Out[8]:'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/'In[9]:sc.get_random_user_agent()Out[9]:'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1'In[10]:sc.get_random_ipv4()Out[10]:'121.162.233.190'In[11]:sc.get_random_ipv4()Out[11]:'178.172.98.169'In[12]:sc.get_random_ipv6()Out[12]:'3d18:cb77:5387:3ee9:1e60:d5f3:d987:283a'In[13]:sc.get_random_ipv6()Out[13]:'cfc1:a00d:9013:37a0:ed94:5e92:7fe7:e356'In[14]:the request headers with user agents will be automatically created.In[1]:# %load examples/check_headers.py...:importscrapinghelperassch...:frompprintimportpprint...:...:scraper=sch.Scraper()...:response=scraper.request('http://httpbin.org/headers')...:...:pprint(response.json()){'headers':{'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9','Accept-Encoding':'gzip, deflate, br','Accept-Language':'en','Host':'httpbin.org','Upgrade-Insecure-Requests':'1','User-Agent':'Mozilla/5.0 (iPad; CPU OS 8_1_2 like Mac OS X) ''AppleWebKit/600.1.4 (KHTML, like Gecko) ''Mobile/12B440','X-Amzn-Trace-Id':'Root=1-62de3626-07daf491262b96356486884d'}}In[2]:**[Important note regardings trie]the datasets of UserAgents from51Degrees/DeviceDetection.
data file '20000 User Agents.csv' is licenced under the European Union Public Licence V.1.2.fromscrapinghelperimportScraperscrapper=Scraper(datapath="path_to_new_user_agent_csvfile")# ...or set shell Environmnet "SCRAPINGHELPER_USERAGENT_PATH".exportSCRAPINGHELPER_USERAGENT_PATH="/path/to/new_user_agent_csvfile"In[2]:fromscrapinghelperimportURL,Scraper,LogConfig...:...:logconfig=LogConfig()...:logconfig.level='INFO'...:sc=Scraper(logconfig=logconfig)...:...:url=URL('https://www.houjin-bangou.nta.go.jp/download/zenken/#csv-unic...:ode')...:response=sc.request(url)...:...:content=response.content...:print(f'code:{response.status_code}')...:code:200In[2]:# %load examples/yahoo_finance.py...:fromtypingimportAny...:importnumpyasnp...:fromscrapinghelperimportScraper,URL...:...:MARKET_SUMMARY=(...:'#market-summary > div > div.D\(ib\).Fl\(start\).W\(100\%\) '...:'> div.Carousel-Mask.Pos\(r\).Ov\(h\).market-summary'...:'.M\(0\).Pos\(r\).Ov\(h\).D\(ib\).Va\(t\) > ul'...:)...:...:classYHFinance(Scraper):...:Finance=URL('https://finance.yahoo.com/')...:...:def__init__(self,**kwargs:Any):...:super().__init__(browser_args='--incognito',**kwargs)...:self.response=self.request(self.Finance.url)...:...:defget_market_summary(self)->list:...:contents=self.get_texts(MARKET_SUMMARY)[0]...:returncontents...:...:if__name__=='__main__':...:fromscrapinghelper.utilsimportsplit_chunks...:...:yahoo=YHFinance()...:summary=yahoo.get_market_summary()...:fordatainsplit_chunks(summary,5):...:print(data)...:['S&P Futures','','4,215.50','-16.00','(-0.38%)']['Dow Futures','','33,583.00','-123.00','(-0.36%)']['Nasdaq Futures','','13,207.00','-61.50','(-0.46%)']['Russell 2000 Futures','','1,952.70','-6.40','(-0.33%)']['Crude Oil','','89.87','-0.90','(-0.99%)']['Gold','','1,759.70','-3.20','(-0.18%)']In[3]:In[3]:fromscrapinghelperimportScraper,LogConfig...:...:logconfig=LogConfig()...:logconfig.level='DEBUG'...:sc=Scraper(logconfig=logconfig)...:...:url=URL('https://www.houjin-bangou.nta.go.jp/download/zenken/#csv-unic...:ode')...:response=sc.request(url)...:...:content=response.content...:print(f'code:{response.status_code}')2022-06-02T19:34:31.885790+0900LOGconfigure:{'handlers':[{'sink':<_io.TextIOWrappername='<stdout>'mode='w'encoding='utf-8'>,'level':'DEBUG','format':'<green>{time}</green> <level>{message}</level>','colorize':True,'serialize':False}]}2022-06-02T19:34:31.886414+0900URL:https://www.houjin-bangou.nta.go.jp/download/zenken/#csv-unicode2022-06-02T14:34:32.092599+0900responsestatus_code:200code:200In[4]:logconfigOut[4]:LogConfig(sink=None,level=DEBUG,format=<green>{time}</green><level>{message}</level>,colorize=True,serialize=FalseIn[5]:render() and PROXYif passedrender=False,request()skip callrender().render()of requests-html does not work with proxy.
scrapinghelper supportrender()with proxy.In[2]:# %load examples/check_ipaddress.py...:fromscrapinghelperimportScraper,ProxyManager,ProxyRotate...:...:# tiny socks5 proxy...:proxies=['socks5://127.0.0.1:9050']...:url='https://httpbin.org/ip'...:...:scraper=Scraper(proxies=proxies)...:...:# default proxy_rotate is ProxyRotate.NO_PROXY...:# does not call render()...:response=scraper.request(url,render=False)...:print(response.html.text)...:...:# using next proxy server....:response=scraper.request(url,proxy_rotate=ProxyRotate.NEXT,render=Fa...:lse)...:print(response.html.text)...:...:# using current proxy server....:response=scraper.request(url,proxy_rotate=ProxyRotate.KEEP,render=Fa...:lse)...:print(response.html.text)...:...:# using random proxy server....:response=scraper.request(url,proxy_rotate=ProxyRotate.RANDOM)...:print(response.html.text)...:{"origin":"221.186.103.38"}{"origin":"185.220.101.182"}{"origin":"185.220.101.182"}{"origin":"185.220.101.182"}In[3]:PROXYGet public proxies list from url.
default isgithub.com/hookzof.
Please keep in mind, there proxies are ABSOLUTELY NO WARRANTY.In[1]:fromscrapinghelperimportProxyManager,PROXYIn[2]:pm=ProxyManager()In[3]:pm.load_proxies('https://raw.githubusercontent.com/hookzof/socks5_list/m...:aster/proxy.txt', proxy_type='socks5')In[4]:pm.proxies[:5]Out[4]:['101.67.215.147:44844','103.110.84.199:7497','103.108.228.185:7497','103.74.121.46:27102','1.57.21.59:7302']In[5]:pm.proxy_poolOut[5]:<itertools.cycleat0x11bdd1b80>In[6]:pm.next_proxy()Out[6]:101.67.215.147:44844In[7]:pm.next_proxy()Out[7]:103.110.84.199:7497In[8]:pm.random_proxy()Out[8]:72.210.252.134:46164In[9]:pm.random_proxy().proxy_mapOut[9]:{'http':'socks5://1.57.21.59:7302','https':'socks5://1.57.21.59:7302'}In[10]:pm.current_proxyOut[10]:1.57.21.59:7302In[11]:pm.current_proxy.attrsOut[11]:{'proxy_url':'1.57.21.59:7302','is_valid':True,'scheme':'socks5','netloc':'1.57.21.59:7302','username':None,'password':None,'hostname':'1.57.21.59','port':7302,'proxy_map':{'http':'socks5://1.57.21.59:7302','https':'socks5://1.57.21.59:7302'}}In[12]:you can filename as url. i.e.:p=ProxyManager('file://./myproxy_list.txt')there uri expand as follows.ifproxies_url.startswith('file://.'):proxies_url=proxies_url.replace('file://.','')this_directory=Path(__file__).parentproxies_url='file://{}/{}'.format(this_directory,proxies_url)CAUTIONif you use a free proxy to login to something or enter personal information and POST it, you must be assured that it will be leaked.
Keep in mind, it is like writing your credit card number and security code on a postcard.KNOWN PROBLEMif you want to use this module(and/or requests_html, selenium) on ubuntu, you should add system libraries. try folloings.sudoaptinstall-ygconf-servicelibasound2libatk1.0-0libc6libcairo2libcups2libdbus-1-3libexpat1libfontconfig1libgcc1libgconf-2-4libgdk-pixbuf2.0-0libglib2.0-0libgtk-3-0libnspr4libpango-1.0-0libpangocairo-1.0-0libstdc++6libx11-6libx11-xcb1libxcb1libxcomposite1libxcursor1libxdamage1libxext6libxfixes3libxi6libxrandr2libxrender1libxss1libxtst6ca-certificatesfonts-liberationlibappindicator1libnss3lsb-releasexdg-utilswgetSee Also:https://techoverflow.net/2020/09/29/how-to-fix-pyppeteer-pyppeteer-errors-browsererror-browser-closed-unexpectedly/