Q:

How to scrape a data from a dynamic website containing Javascript using Python? 

I am trying to scrape data from https://www.doordash.com/food-delivery/chicago-il-restaurants/
The idea is to scrape all the data regarding the different restaurant listings on the website.  The site is divided into different cities, but I only require restaurant data for Chicago.  
All restaurant listings for the city have to be scraped along with any other relevant data about the respective restaurants (Ex: Reviews, Rating, Cuisine, address, state etc).  I need to capture all the respective details(currently 4,326 listings) for the city in the Excel.  
I have tried to extract the restaurant name, cuisine, ratings and review inside the class named "StoreCard_root___1p3uN".  But No datas have been displayed.  The output is blank. 

from selenium import webdriver

chrome_path = r"D:\python project\chromedriver.exe"

driver = webdriver. Chrome(chrome_path)

driver.get("https://www.doordash.com/food-delivery/chicago-il-restaurants/")

driver.find_element_by_xpath("""//*[@id="SeoApp"]/div/div[1]/div/div[2]/div/div[2]/div/div[2]/div[1]/div[3]""").click()

posts = driver.find_elements_by_class_name("StoreCard_root___1p3uN")

for post in posts:
    print(post.text) ```

A:

you can use the API url as the data rendered from it actually via XHR request. 
iterate over the API link below and scrape whatever you want. 
https://api.doordash.com/v2/seo_city_stores/?delivery_city_slug=chicago-il-restaurants&store_only=true&limit=50&offset=0
You will just loop over this parameter offset=0 by increasing it +50 each time as each page will shown 50 items till you reach 4300 as it's the last page !  simply by range(0, 4350, 50)
import requests
import pandas as pd

data = []
for item in range(0, 4350, 50):
    print(f"Extracting item# {item}")
    r = requests.get(
        f"https://api.doordash.com/v2/seo_city_stores/?delivery_city_slug=chicago-il-restaurants&store_only=true&limit=50&offset={item}").json()
    for item in r['store_data']:
        item = (item['name'], item['city'], item['category'],
                item['num_ratings'], item['average_rating'], item['average_cost'])
        data.append(item)

df = pd. DataFrame(
    data, columns=['Name', 'City', 'Category', 'Num Ratings', 'Average Ratings', 'Average Cost'])
df.to_csv('output.csv', index=False)
print("done")

Sample of Output:

View Output online: Click Here
Full Data is here: Click Here

