Package: ecoindex_scraper
Ecoindex ScraperThis module provides a simple interface to get theEcoindexof a given webpage using moduleecoindex-computeRequirementsPython ^3.10 withpipInstallpipinstallecoindex-scraperUseGet a page analysisYou can run a page analysis by calling the functionget_page_analysis():(function)get_page_analysis:(url:AnyHttpUrl,window_size:WindowSize|None=WindowSize(width=1920,height=1080),wait_before_scroll:int|None=1,wait_after_scroll:int|None=1)->Coroutine[Any,Any,Result]Example:importasynciofrompprintimportpprintfromecoindex.scrapimportEcoindexScraperpprint(asyncio.run(EcoindexScraper(url="http://ecoindex.fr").get_page_analysis()))Result example:Result(width=1920,height=1080,url=AnyHttpUrl('http://ecoindex.fr',),size=549.253,nodes=52,requests=12,grade='A',score=90.0,ges=1.2,water=1.8,ecoindex_version='5.0.0',date=datetime.datetime(2022,9,12,10,54,46,773443),page_type=None)Default behaviour:By default, the page analysis simulates:Window size of1920x1080pixels (can be set with parameterwindow_size)Wait for1 second when page is loaded(can be set with parameterwait_before_scroll)Scroll to the bottom of the page (if it is possible)Wait for1 second afterhaving scrolled to the bottom of the page (can be set with parameterwait_after_scroll)Get a page analysis and generate a screenshotIt is possible to generate a screenshot of the analyzed page by adding aScreenShotproperty to theEcoindexScraperobject.
You have to define an id (can be a string, but it is recommended to use a unique id) and a path to the screenshot file (if the folder does not exist, it will be created).importasynciofrompprintimportpprintfromuuidimportuuid1fromecoindex.modelsimportScreenShotfromecoindex.scrapimportEcoindexScraperpprint(asyncio.run(EcoindexScraper(url="http://www.ecoindex.fr/",screenshot=ScreenShot(id=str(uuid1()),folder="./screenshots"),).get_page_analysis()))Async analysisYou can also run the analysis asynchronously:importasynciofromconcurrent.futuresimportThreadPoolExecutor,as_completedfromecoindex.scrapimportEcoindexScraperdefrun_page_analysis(url):returnasyncio.run(EcoindexScraper(url=url).get_page_analysis())withThreadPoolExecutor(max_workers=8)asexecutor:future_to_analysis={}url="https://www.ecoindex.fr"foriinrange(10):future_to_analysis[executor.submit(run_page_analysis,url,)]=(url)forfutureinas_completed(future_to_analysis):try:print(future.result())exceptExceptionase:print(e)Get requests details from an analysisYou can get the details of the requests made by the page by calling the functionget_all_requests()and also get the aggregation of requests by category by calling the functionget_requests_by_category():importasynciofrompprintimportpprintfromecoindex.scraperimportEcoindexScraperscraper=EcoindexScraper(url="http://www.ecoindex.fr")result=asyncio.run(scraper.get_page_analysis())all_requests=asyncio.run(scraper.get_all_requests())requests_by_category=asyncio.run(scraper.get_requests_by_category())pprint([request.model_dump()forrequestinall_requests])# [{'category': 'html',#   'mime_type': 'text/html; charset=iso-8859-1',#   'size': 475.0,#   'status': 301,#   'url': 'http://www.ecoindex.fr/'},#  {'category': 'html',#   'mime_type': 'text/html',#   'size': 7772.0,#   'status': 200,#   'url': 'https://www.ecoindex.fr/'},#  {'category': 'css',#   'mime_type': 'text/css',#   'size': 9631.0,#   'status': 200,#   'url': 'https://www.ecoindex.fr/css/bundle.min.d38033feecefa0352173204171412aec01f58eee728df0ac5c917a396ca0bc14.css'},#  {'category': 'javascript',#   'mime_type': 'application/javascript',#   'size': 9823.0,#   'status': 200,#   'url': 'https://www.ecoindex.fr/fr/js/bundle.8781a9ae8d87b4ebaa689167fc17b7d71193cf514eb8bb40aac9bf4548e14533.js'},#  {'category': 'other',#   'mime_type': 'x-unknown',#   'size': 892.0,#   'status': 200,#   'url': 'https://www.ecoindex.fr/images/logo-neutral-it.webp'},#  {'category': 'image',#   'mime_type': 'image/svg+xml',#   'size': 3298.0,#   'status': 200,#   'url': 'https://www.ecoindex.fr/images/logo-greenit.svg'}]pprint(requests_by_category.model_dump())# {'css': {'total_count': 1, 'total_size': 9631.0},#  'font': {'total_count': 0, 'total_size': 0.0},#  'html': {'total_count': 2, 'total_size': 8247.0},#  'image': {'total_count': 1, 'total_size': 3298.0},#  'javascript': {'total_count': 1, 'total_size': 9823.0},#  'other': {'total_count': 1, 'total_size': 892.0},#  'video': {'total_count': 0, 'total_size': 0.0}}