Package: shopscraper
ShopScraperShopScraper is a thin python wrapper for Shopify webshop product APIs used to scrape information from online stores. Every Shopify webshop has a "hidden" api with access to all of the store's products. This python package uses the requests library to grab the product information and return objects representing each product. There are also convenience functions for saving to and reading from a JSON file.>>>importshopscraper>>>>>>products=shopscraper.scrape("bjjfanatics.com",include_html=False,items_per_page=2,max_pages=1)>>>type(products)<class'generator'>>>>forproductinproducts:>>>print(product)id=6706690981986,title='New Wave Jiu Jitsu: Side Attacks - Building a Devastating Side Control System by John Danaher'handle='new-wave-jiu-jitsu-side-attacks-building-a-devastating-side-control-system-by-john-danaher',body_html='',published_at=datetime.datetime(2022,5,18,8,2,9,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),created_at=datetime.datetime(2022,5,4,23,14,55,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),updated_at=datetime.datetime(2022,5,18,11,55,59,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),vendor='John Danaher',product_type='COMBO',tags=['Facebook','Fighter_John Danaher','MC_Side_Control_Attacks','New','new_and_popular','Show_More_App'],variants=[Variant(id=39769726582882,title='Default Title',option1='Default Title',option2=None,option3=None,sku='JDNWJJSA-01',requires_shipping=False,taxable=True,featured_image=None,available=True,price=19700,grams=0,compare_at_price=None,position=1,product_id=6706690981986,created_at=datetime.datetime(2022,5,4,23,14,55,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),updated_at=datetime.datetime(2022,5,18,11,55,15,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))))]images=[Image(id=28542700257378,created_at=datetime.datetime(2022,5,4,23,14,55,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),position=1,updated_at=datetime.datetime(2022,5,4,23,14,57,tzinfo=datetime.timezone(datetime.timedelta(days=-1,seconds=72000))),product_id=6706690981986,variant_ids=[],src='https://cdn.shopify.com/s/files/1/1800/2299/products/JohnDanaher_NewWaveJiu-Jitsu-SideAttacks_CoverFRONT.jpg?v=1651720497',width=1631,height=2194),]options=[Options(name='Title',position=1,values=['Default Title'])]InstallationShopScraper is available on PyPI:$python-mpipinstallshopscraperShopScraper officially supports Python 3.9+.UsageThe 'scrape' and 'read_json' functions yield product objects with lists of Variant, Image, and Options objects.classImage:"""Attributes:id (int): image idcreated_at (datetime.datetime): datetime object of when image was createdposition (int): position of image in productupdated_at (datetime.datetime): datetime object of when image was last updatedproduct_id (int): product id associated with the imagevariant_ids (list[int]): list of variant ids associated with the imagesrc (str): url to imagewidth (int): width of image in pixelsheight (int): height of image in pixels"""classOptions:"""Attributes:name (str): name of optionposition (int): position of option in productvalues (list[Any]): list of values for option"""classVariant:"""Attributes:id (int): variant idtitle (str): title of variantoption1 (str): first option of variantoption2 (str): second option of variantoption3 (str): third option of variantsku (Optional[str]): sku of variantrequires_shipping (bool): whether variant requires shippingtaxable (bool): whether variant is taxablefeatured_image (Image): featured image of variantprice (float): price of variantgrams (int): weight of variant in gramscompare_at_price (Optional(float)): compare at price of variantposition (int): position of variant in productproduct_id (int): product id associated with the variantcreated_at (datetime.datetime): datetime object of when variant was createdupdated_at (datetime.datetime): datetime object of when variant was last updated"""classProduct:"""Attributes:id (int): product idtitle (str): name of producthandle (str): url safe name of productbody_html (str): description of product (html)published_at (datetime.datetime): date product was publishedcreated_at (datetime.datetime): date product was createdupdated_at (datetime.datetime): date product was last updatedvendor (str): name of product vendorproduct_type (str): type of producttags (list[str]): tags associated with productvariants (list[Variants]): list of variants for productimages (list[Image]): list of images for productoptions (list[Options]): list of options for product"""The 'scrape' function yields product objects:>>>importshopscraper>>>>>>products=shopscraper.scrape("bjjfanatics.com")>>>type(products)<class'generator'>The 'scrape_to_json' function saves the scraped data to the specified file path:>>>importshopscraper>>>>>>save_path=shopscraper.scrape_to_json("bjjfanatics.com","C:\\scraped_data.json")>>>type(save_path)<class'pathlib.Path'>The 'read_json' function reads the saved json file and yields Product objects:>>>importshopscraper>>>>>>products=shopscraper.read_json("C:\\scraped_data.json")>>>type(products)<class'generator'>Note that both functions that provide product objects are generators, which are more memory efficient than lists but can only be iterated over one time. If you want to use the product objects in more than one operation, accumulate them into a list:>>>importshopscraper>>>>>>products=shopscraper.read_json("C:\\scraped_data.json")>>>product_list=list(products)>>>len(product_list)200>>>combo_products=[iforiinproduct_listifi.product_type=="COMBO"]>>>len(combo_products)10