Scraping data in network traffic using Python

13 Jul, 2021
In this article, we will learn how to scrap data in network traffic using Python.
selenium: Selenium is a portable framework for controlling web browser.
time: This module provides various time-related functions.
json: This module is required to work with JSON data.
browsermobproxy: This module helps us to get the HAR file from network traffic.
There are two ways by which we can scrap the network traffic data.
To start with this download and extract the chrome webdriver from here according to the version of your chrome browser and copy the executable path.
Approach:
Import the DesiredCapabilities from the selenium module and enable performance logging.
Startup the chrome webdriver with executable_path and default chrome-options or add some arguments to it and the modified desired_capabilities.
Send a GET request to the website using driver.get() and wait for few seconds to load the page.
Syntax:
driver.get(url)
Get the performance logs using driver.get_log() and store it in a variable.
Syntax:
driver.get_log(“performance”)
Iterate every log and parse it using json.loads() to filter all the Network related logs.
Write the filtered logs to a JSON file by converting to JSON string using json.dumps().
Example:
Python3
# Import the required modulesfrom selenium import webdriverfrom selenium.webdriver.common.desired_capabilities import DesiredCapabilitiesimport timeimport json    # Main Functionif __name__ == "__main__":      # Enable Performance Logging of Chrome.    desired_capabilities = DesiredCapabilities.CHROME    desired_capabilities["goog:loggingPrefs"] = {"performance": "ALL"}      # Create the webdriver object and pass the arguments    options = webdriver.ChromeOptions()      # Chrome will start in Headless mode    options.add_argument('headless')      # Ignores any certificate errors if there is any    options.add_argument("--ignore-certificate-errors")      # Startup the chrome webdriver with executable path and    # pass the chrome options and desired capabilities as    # parameters.    driver = webdriver.Chrome(executable_path="C:/chromedriver.exe",                              chrome_options=options,                              desired_capabilities=desired_capabilities)      # Send a request to the website and let it load    driver.get("https://www.geeksforgeeks.org/")      # Sleeps for 10 seconds    time.sleep(10)      # Gets all the logs from performance in Chrome    logs = driver.get_log("performance")      # Opens a writable JSON file and writes the logs in it    with open("network_log.json", "w", encoding="utf-8") as f:        f.write("[")          # Iterates every logs and parses it using JSON        for log in logs:            network_log = json.loads(log["message"])["message"]              # Checks if the current 'method' key has any            # Network related value.            if("Network.response" in network_log["method"]                    or "Network.request" in network_log["method"]                    or "Network.webSocket" in network_log["method"]):                  # Writes the network log to a JSON file by                # converting the dictionary to a JSON string                # using json.dumps().                f.write(json.dumps(network_log)+",")        f.write("{}]")      print("Quitting Selenium WebDriver")    driver.quit()      # Read the JSON File and parse it using    # json.loads() to find the urls containing images.    json_file_path = "network_log.json"    with open(json_file_path, "r", encoding="utf-8") as f:        logs = json.loads(f.read())      # Iterate the logs    for log in logs:          # Except block will be accessed if any of the        # following keys are missing.        try:            # URL is present inside the following keys            url = log["params"]["request"]["url"]              # Checks if the extension is .png or .jpg            if url[len(url)-4:] == ".png" or url[len(url)-4:] == ".jpg":                print(url, end='\n\n')        except Exception as e:            pass
Output:
The image URL’s are highlighted above.
network_log.json containing the image URL’s
For this, the following requirements need to be satisfied.
Download and Install Java v8 from here
Download and extract browsermobproxy from here and copy the path of bin folder.
Install browsermob-proxy using pip using the command in terminal : 
pip install browsermob-proxy
Download and extract the chrome webdriver from here, according the version of your chrome browser and copy the executable path.
Approach:
Import the Server module from browsermobproxy and start up the Server with the copied bin folder path and set port as 8090.
Call the create_proxy method to create the proxy object from Server and set “trustAllServers” parameter as true.
Startup the chrome webdriver with executable_path and chrome-options discussed in code below.
Now, create a new HAR file using the proxy object with the domain of the website.
Send a GET request using driver.get() and wait for few seconds to load it properly.
Syntax:
driver.get(url)
Write the HAR file of network traffic from the proxy object to a HAR file by converting it to JSON string using json.dumps().
Example:
Python3
# Import the required modulesfrom selenium import webdriverfrom browsermobproxy import Serverimport timeimport json    # Main Functionif __name__ == "__main__":      # Enter the path of bin folder by    # extracting browsermob-proxy-2.1.4-bin    path_to_browsermobproxy = "C:\\browsermob-proxy-2.1.4\\bin\\"      # Start the server with the path and port 8090    server = Server(path_to_browsermobproxy                    + "browsermob-proxy", options={'port': 8090})    server.start()      # Create the proxy with following parameter as true    proxy = server.create_proxy(params={"trustAllServers": "true"})      # Create the webdriver object and pass the arguments    options = webdriver.ChromeOptions()      # Chrome will start in Headless mode    options.add_argument('headless')      # Ignores any certificate errors if there is any    options.add_argument("--ignore-certificate-errors")      # Setting up Proxy for chrome    options.add_argument("--proxy-server={0}".format(proxy.proxy))      # Startup the chrome webdriver with executable path and    # the chrome options as parameters.    driver = webdriver.Chrome(executable_path="C:/chromedriver.exe",                              chrome_options=options)      # Create a new HAR file of the following domain    # using the proxy.    proxy.new_har("geeksforgeeks.org/")      # Send a request to the website and let it load    driver.get("https://www.geeksforgeeks.org/")      # Sleeps for 10 seconds    time.sleep(10)      # Write it to a HAR file.    with open("network_log1.har", "w", encoding="utf-8") as f:        f.write(json.dumps(proxy.har))      print("Quitting Selenium WebDriver")    driver.quit()      # Read HAR File and parse it using JSON    # to find the urls containing images.    har_file_path = "network_log1.har"    with open(har_file_path, "r", encoding="utf-8") as f:        logs = json.loads(f.read())      # Store the network logs from 'entries' key and    # iterate them    network_logs = logs['log']['entries']    for log in network_logs:          # Except block will be accessed if any of the        # following keys are missing        try:            # URL is present inside the following keys            url = log['request']['url']              # Checks if the extension is .png or .jpg            if url[len(url)-4:] == '.png' or url[len(url)-4:] == '.jpg':                print(url, end="\n\n")        except Exception as e:            # print(e)            pass
Output:
The image URL’s are highlighted above.
network_log1.har containing the image URL’s
Picked
Python-selenium
Web-scraping
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
How to drop one or multiple columns in Pandas Dataframe
Python | os.path.join() method
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python | Get unique values from a list
Python | datetime.timedelta() function