WebScraping but no browser network traffic: Peaking at JavaScript
Channel: AUTOHOTKEY Gurus
Date: 2021-07-19

You might have loaded a web page and then noticed that, even though you interact with buttons, there is no visible network traffic.  Often this will be because the page has cached JavaScript.  Here RaptorX and the-Automator walk through how to look for eventlisteners and find the Javascript that was downloaded

TLDR: The video discusses how to detect and work around JavaScript web scraping without an API by using the network tab and browser's text conversion feature.

1. 00:00 🔍 Detecting and working around JavaScript web scraping without an API by using the network tab and browser's text conversion feature.
 1.1 The lecture discusses how to detect when JavaScript is being used for web scraping and how to work around situations where there is no API available.
 1.2 Pressing the f12 key and accessing the network tab provides useful information about requests and responses when accessing a website.
 1.3 Browser is catching and converting typed text into a list without making any requests to the server.

2. 03:46 🔍 To find out what JavaScript is running on a website, inspect the event listeners to see the contents of the file being loaded.
 2.1 To figure out what JavaScript is running locally on a website, go to the control where the event is happening, hit inspect, and look for the event listeners, which will show you the contents of the JavaScript file being loaded onto the website.
 2.2 To detect and understand the type of JavaScript being executed on a webpage, check for the presence of JavaScript files, which may be separated by functions on larger sites.
 2.3 The search function in the JavaScript code is executed when the keyboard is released, and it builds up a list of search results.

3. 07:42 👨‍💻 Understanding JavaScript can help locate specific data in a program and access cache information.
 3.1 Understanding JavaScript and coding languages can help identify variables and functions in order to extract specific information, such as a search list.
 3.2 The speaker discusses how to locate a specific file and access cache data in a program, and mentions the functions involved in creating and setting up data in HTML.
 3.3 The speaker discussed the location of a function and a for loop in code.

4. 12:06 🔍 A search function was improved by examining JavaScript files and headers on the website.
 4.1 A search function was created to rank and display results in order of relevance based on an index, with the URL being the key.
 4.2 The speaker found the necessary information by examining the JavaScript files and headers on the website.

5. 15:38 🔍 Use network traffic to find original files on websites, but added header files may not be visible.
 5.1 Accessing information on websites can be difficult, but using the function of watching sources can help locate the original file.
 5.2 The speaker explains how to view network traffic to see the initial browser call and API call, but notes that the added header file will not be visible in the network traffic.

6. 18:12 📝 The website doesn't have local cache, so every click creates a new request.
 6.1 The website's scripts may not be loaded if they are already stored locally, and clearing the cache may help, but the site's storage is zero and does not have application cache.
 6.2 The special header loads the HTML into memory on the first load, allowing the browser to read the crypts without showing up as a network.
 6.3 The webpage doesn't have local cache, so every time you click on something, it creates a new request even if you have already visited the page.

7. 21:23 🕸️ Web scraping can be complex, but using API calls is often better than navigating through a browser, although the Win HTTP request object can retrieve HTML and JavaScript files.
 7.1 The Win HTTP request object cannot execute JavaScript, but it can retrieve HTML and JavaScript files which can be manually executed.
 7.2 The lecture discusses the complexities of web scraping and highlights the benefits of using API calls instead, while also noting that in some cases navigating through a browser can be a viable alternative.

8. 24:00 📺 The latest video on the speaker's YouTube channel discusses the impact of social media on mental health.
 8.1 Subscribe to the speaker's YouTube channel and turn on all notifications to stay updated.


Summary for https://youtu.be/LSmyVcHSipY by www.eightify.app