How to Build a Web Scraper in Python | by Roman Paolucci | Towards Data Science

Web scraping is an awesome tool for analysts to sift through and collect large amounts of public data. Using keywords relevant to the topic in question, a good web scraper can gather large amounts of data very quickly and aggregate it into a dataset. There are several libraries in Python that make this extremely easy to accomplish. In this article, I will illustrate an architecture that I have been using for web scraping and summarizing search engine data. The article will be broken up into the following sections...
Link Scraping
Content Scraping
Content Summarizing
Building a Pipeline
All of the code will be provided herein.
First, we need a way to gather URLs relevant to the topic we are scraping data for. Fortunately, the Python library googlesearch makes it easy to gather URLs in response to an initial google search. Let’s build a class that uses this library to search our keywords and append a fixed number of URLs to a list for further analysis...
This is arguably the most important part of the web scraper as it determines what data on a webpage will be gathered. Using a combination of urllib and beautiful soup (bs4) we are able to retrieve and parse the HTML for each URL in our Link Scraper class. Beautiful soup lets us specify the tags we want to extract data from. In the case below I am establishing a URL request and parsing the HTML response with bs4 and storing all the information found in the paragraph (<p></p>) tags...
This is where we create a summary of the text extracted from each page’s HTML residing in our Content Scraper. To do this we will be using a combination of libraries, mainly NLTK. The way in which we are generating the summary is relatively elementary and there are many ways to improve this method — but it's a great start. After some formatting and voiding of filler words, words get tokenized and ranked by frequency generating a few sentences that aim to accurately summarize the article...
This is the part where we put everything together. One class will instantiate an instance of each other component as needed to build and implement our web scraper. The WebScraper class takes a few parameters...
search — String, Search engine query
n — Integer, Number of URL sources to analyze
sl —Integer, Sentence length of the summary
fall_through — Boolean, Multithread the process or not
Write_file —Boolean, Write the summaries to a file
Let’s now instantiate and run an instance of this WebScraper class...
Analyst(‘AAPL’, 10, 3, False, False)
Running the previous code results in the following output...
Analyzing:  http://t1.gstatic.com/images?q=tbn:ANd9GcSjoU2lZ2eJX3aCMfiFDt39uRNcDu9W7pTKcyZymE2iKa7IOVaIAnalyzing:  https://en.wikipedia.org/wiki/Apple_Inc.Analyzing:  https://www.bloomberg.com/news/articles/2020-08-26/apple-plans-augmented-reality-content-to-boost-tv-video-serviceAnalyzing:  https://www.marketwatch.com/story/apple-stock-rises-after-wedbush-hikes-target-to-new-street-high-of-600-2020-08-26Analyzing:  https://www.marketwatch.com/story/tesla-and-apple-have-had-a-great-run-heres-why-theyre-poised-to-rocket-even-higher-in-the-next-year-2020-08-26Analyzing:  https://finance.yahoo.com/quote/AAPL/Analyzing:  https://seekingalpha.com/article/4370830-apple-sees-extreme-bullishnessAnalyzing:  https://seekingalpha.com/news/3608898-apples-newest-street-high-price-target-700-bull-caseAnalyzing:  https://www.marketwatch.com/investing/stock/aaplAnalyzing:  https://stocktwits.com/symbol/AAPLencoding error : input conversion failed due to input error, bytes 0x9D 0x09 0x96 0xA3encoding error : input conversion failed due to input error, bytes 0x9D 0x09 0x96 0xA3Value Error***For more information you can review our Terms of Service and Cookie Policy.For inquiries related to this message please contact our support team and provide the reference ID below.***URL ErrorValue Error"China remains a key ingredient in Apple's recipe for success as we estimate roughly 20% of iPhone upgrades will be coming from this region over the coming year."Ives points to recent signs of momentum in China, which he expects will continue for the next six to nine months.Real-time last sale data for U.S. stock quotes reflect trades reported through Nasdaq only.***By comparison, Amazon AMZN, +0.58% has split its stock three times, rallying an average of 209% the following year.Apple�s history isn�t quite as stellar as all those, with its four previous splits resulting in an average gain of 10.4% in the following year.Real-time last sale data for U.S. stock quotes reflect trades reported through Nasdaq only.***The truck and fuel cell maker �could be a major horse in the EV race,� wrote the analyst, while voicing concerns about the stock�s valuation.stocks edged lower Wednesday, a day after the S&P 500 set its first record close since February, after Federal Reserve officials highlighted the uncertainties facing the economy.stock-market benchmarks mostly opened higher on Wednesday, pushing the key benchmarks to further records after an economic report came in better than expected.***Apple is the world's largest information technology company by revenue, the world's largest technology company by total assets, and the world's second-largest mobile phone manufacturer after Samsung.Two million iPhones were sold in the first twenty-four hours of pre-ordering and over five million handsets were sold in the first three days of its launch.The same year, Apple introduced System 7, a major upgrade to the operating system which added color to the interface and introduced new networking capabilities.***Howley also weighs in on Nintendo potentially releasing an upgraded Switch in 2021.***[Finished in 5.443s]
We have successfully extracted a few summaries from top search results about AAPL. Some sites have this type of request blocked as seen in the console output. Nevertheless, this has been a comprehensive starter guide to web scraping in Python.