Quickly extract all links from a web page using the browser console | by Phil Gorman | Towards Data Science

Extracting and cleaning data from websites and documents is my bread and butter and I have really enjoyed learning how to systematically extract data from multiple web pages and even multiple websites using Python and R. But sometimes a project only needs a small amount of data, from just one page on a website.
Previously when a case like this arose, I would still fire up my Python IDE or RStudio, write and execute a script to extract this information. This is using a sledgehammer to crack a nut. Regular old JavaScript is powerful enough to extract information from a single web page, and the JavaScript in question can be run in the browser’s developer console.
In this example, I am extracting all links from a web page, as this is a task I regularly perform on web pages. However, this code would work equally well to extract any other text element types in HTML documents, with a few small changes. When this code runs, it opens a new tab in the browser and outputs a table containing the text of each hyperlink and the link itself, so there is some context to what each link is pointing to.
Literally any browser made in the past 10 years.
The bit of code I’ll be providing further down the page.
That’s it!
Open up your browser (yes, this even works in Internet Explorer if you’re a glutton for punishment) and navigate to the page from which you’d like to extract links.
I’m using the Select Committee inquiries list from the 2017 Parliament page as an example — it is a page with a massive amount of links that, as a grouping, may be useful to a lot of people.
Now we just need to open up the developer console and run the code. To open the developer console, you can either hit F12 or right-click the page and select “Inspect” or ”Inspect Element” depending on your browser of choice. This will open up the console, into which you can type or copy and paste snippets of code.
This is the code snippet you will need to place into the console:
var x = document.querySelectorAll("a");var myarray = []for (var i=0; i<x.length; i++){var nametext = x[i].textContent;var cleantext = nametext.replace(/\s+/g, ' ').trim();var cleanlink = x[i].href;myarray.push([cleantext,cleanlink]);};function make_table() {    var table = '<table><thead><th>Name</th><th>Links</th></thead><tbody>';   for (var i=0; i<myarray.length; i++) {            table += '<tr><td>'+ myarray[i][0] + '</td><td>'+myarray[i][1]+'</td></tr>';    };     var w = window.open("");w.document.write(table); }make_table()
Then just hit enter (or the run button in IE. But really, why are you using IE?!). This will open up a new tab in your browser with a table containing all the link text and hyperlinks from your chosen web page.
This table can then be copied and pasted into a spreadsheet or document to be used as you please.
Here is a breakdown of the code and what each aspect does.
var x = document.querySelectorAll("a");var myarray = []
Here we are finding all of the “a” elements on the page (a elements are links) and assigning them to the variable x. Then we create an array variable, but leave it empty for now.
for (var i=0; i<x.length; i++){var nametext = x[i].textContent;var cleantext = nametext.replace(/\s+/g, ' ').trim();var cleanlink = x[i].href;myarray.push([cleantext,cleanlink]);};
We then loop over all the “a” elements in x, and for each element, we try to find the text content of the element and the link.
For the text content, we replace the white space with single spaces and trim the text, as there may be large swathes of white space that would make our table unreadable.
function make_table() {    var table = '<table><thead><th>Name</th><th>Links</th></thead><tbody>';   for (var i=0; i<myarray.length; i++) {            table += '<tr><td>'+ myarray[i][0] + '</td><td>'+myarray[i][1]+'</td></tr>';    };     var w = window.open("");w.document.write(table); }make_table()
We then make the table using the function “make_table()”. This creates a variable, table, with the beginnings of a HTML table and the table headers. We then use a for loop to add table rows containing the link text and hyperlinks.
We then open a new window using “window.open()” and write the HTML table to that window using “document.write()”.
There is a drawback to the current code — it will take ALL of the links on a page. This means all the links in the menus, any jump links that take you to different points on the current page, the contact, T&Cs, sitemap links at the base of the page, etc...
You could be more specific and look for all “a” elements within a certain area of the web page. In the case of the committee inquiries page above we just need to change the first line of code to make the querySelector look at a specific area of the page — perhaps just the a-z listing in the centre of the page.
To do this, right-click the area of the page from which you’d like to take the links and click “Inspect”. You should be able to then see what that element of the page is called. In this case, the element is a “div” with the class “a-to-z-listing”. We can simply change the first line to look for all “a” elements in the element with class “a-to-z-listing”:
var x = document.querySelectorAll(".a-to-z-listing a");
I’m still going to use Python and R in the vast majority of my web scraping activity, but it is helpful to have a quick and easy way to extract information from a web page without the need to open any other applications.
What would you use this code for? Is there a more elegant way to construct this code? Please let me know in the responses below.