Question: My aim is to web-scrape the table on https://data.eastmoney.com/executive/list.html and save it to an excel. Please note that it has 2945 pages and I want to put all of them into one excel sheet. The easiest way of doing it is to see where the data is pulled. So I F12 the webpage to get the source code and saw a data center <datacenter-web.eastmoney.com>  So I use that API within the red box above to get the data. Here's my code import pandas as pd import requests df = pd.DataFrame( requests.get('https://datacenter-web.eastmoney.com/api/data/v1/get? reportName=RPT_EXECUTIVE_HOLD_DETAILS&columns=ALL&sortColumns=CHANGE_DATE')\ .json().get('result').get('data')) df.to_excel('G:/ExecutiveHoldings/all_by_date.xlsx', index = True)  I want to get all the columns and the row to be sorted by CHANGE_DATE, and here's the data I get:  Note that the rows only have a limited number of 500 lines, index from 0 ~ 499. Yet I want to get all dataframe from the web, with line numbers way beyond 500. Is there any easy method to get all 2945 pages of table and put them all into one excel sheet?

Answer: You can retrieve the next page using the pageNumber={page_number} parameter. The pageSize parameter determines how many data items are retrieved per request. The default and maximum value is 500. This is the access URL https://datacenter-web.eastmoney.com/api/data/v1/get?reportName=RPT_EXECUTIVE_HOLD_DETAILS&columns=ALL&sortColumns=CHANGE_DATE&pageNumber={page_number}&pageSize=500  This code can get all of rows data. The script fetches paginated data from a specified URL until no more data is available. Data is collected in the all_data list, with each page containing up to 500 items. Progress, including page number and data count, is logged during each fetch. After fetching all data, it is converted into a pandas DataFrame. The DataFrame is saved to an Excel file named all_by_date.xlsx. Key Logic  While Loop Condition: The loop will continue indefinitely until a break statement is encountered. This is because the condition for the while loop is always True  fetch-data.py import requests import pandas as pd def fetch_data(): all_data = [] page_number = 1 while True: url = f'https://datacenter-web.eastmoney.com/api/data/v1/get?reportName=RPT_EXECUTIVE_HOLD_DETAILS&columns=ALL&sortColumns=CHANGE_DATE&pageNumber={page_number}&pageSize=500' try: response = requests.get(url) response.raise_for_status() result = response.json().get('result', {}) if not result or not result.get('data') or len(result['data']) == 0: break # Exit loop if data array is empty all_data.extend(result['data']) print(f'Page: {page_number}, Data Count: {len(result["data"])}, Total Data: {len(all_data)}') page_number += 1 except requests.exceptions.RequestException as e: print(f'Error fetching data from page {page_number}:', e) break # Exit loop on error return all_data def save_data_to_excel(data): df = pd.DataFrame(data) df.to_excel('./all_by_date.xlsx', index=False) print('Data successfully saved to all_by_date.xlsx') if __name__ == '__main__': data = fetch_data() if data: save_data_to_excel(data) else: print('No data fetched.')  Install dependencies & Run it pip install requests pandas openpyxl python fetch-data.py  Total Data 50 * 2944 + 39 = 147,239 data *Note last on page 2945 has 39 rows Result  147,240 rows - header row = 147,239 row