Web scraper for extracting emails based on keywords and regions - GeeksforGeeks

29 Dec, 2020
Web scraping is a task that is normally performed to scrape structured data from the websites which are then stored accordingly, this kind of data is valuable enough to open the doors to a variety of fields from data mining-related stuff to the data science related applications where large amounts of data are required to make business decisions.
And as for this article, we are going to discuss how to use web scrapers for extracting emails based on keywords and locations.
So the question arises, why would we need something like that? Well, emails extracted based on specific topics and regions can be a very productive way of doing advertisement and product promotion, though one would say that this could be used for black hat SEO it actually depends on how you use it.
Scrapy module: It is used as a Python framework for web scrapping. Getting data from a normal website is easier, and can be just achieved by just pulling HTML of the website and fetching data by filtering tags. It can be installed using the below command.
pip install scrapy
Selenium module: It is a powerful tool for controlling a web browser through the program. It is functional for all browsers, works on all major OS and its scripts are written in various languages i.e Python, Java, C#, etc. It can be installed using the below command.
pip install selenium
Scrapy-Selenium module: It is a scrapy middleware to handle JavaScript pages using selenium. It can be installed using the below command.
pip install scrapy-selenium
Google module: Using python package google we can get the result of google search from a python script. It can be installed using the below command.
pip install google
Step 1: Creating scrapy project with the below command: 
scrapy startproject email_extraction
After executing the above command you will see a folder with the tree like this 
├── email_extraction
│   ├── __init__.py
│   ├── items.py
│   ├── middlewares.py
│   ├── pipelines.py
│   ├── __pycache__
│   ├── settings.py
│   └── spiders
│       ├── email_extractor.py
│       ├── __init__.py
│       └── __pycache__
└── scrapy.cfg

4 directories, 8 files
Create a python file in the spiders directory and open it up in any editor.
Step 2: Importing the required libraries  
Python3
# import required modulesimport scrapyfrom scrapy.spiders import CrawlSpider, Requestfrom googlesearch import searchimport refrom scrapy_selenium import SeleniumRequestfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as EC
  Now the required libraries have been imported we can get to the next step of our script.
 Step 3: Setting up required parameters for the crawler 
Python3
# create class to extract email idsclass email_extractor(CrawlSpider):         # adjusting parameters    name = 'email_ex'     def __init__(self, *args, **kwargs):        super(email_extractor, self).__init__(*args, **kwargs)        self.email_list = []        self.query = " 'market places'.gmail.com "
  In this we are creating class email_extractor and inheriting CrawlSpider class from the scrapy module, in the next line we are giving a unique name to our crawler which we will use later to run our spider, we don’t need to set allowed domain parameter as we will be jumping from one website domain to the other for extracting emails then we are creating a python constructor and declaring a list and a string variable, the string value given here is going to be fed to the google search engine which is our actual query defining keyword (health), location (usa) and .gmail.com for getting email oriented search results.
 Step 4: Getting results and sending requests 
Python3
# sending requests    def start_requests(self):        for results in search(self.query, num=10, stop=None, pause=2):            yield SeleniumRequest(                url=results,                callback=self.parse,                wait_until=EC.presence_of_element_located(                    (By.TAG_NAME, "html")),                dont_filter=True            )
  Here, we are creating our method start_requests then we are using search() method from googlesearch module with parameters of  
query variable which has the actual search query that we declared before.
10 results after every 2 seconds of pause.
We are getting all the results there exist for the query with the stop parameter set to None.
After that, we are yielding a request with the method SeleniumRequest  from scrapy_selenium module with parameters of:  
Getting the first URL sequentially from the search results.
Calling back a method for each URL for further processing which we will see in a minute.
Using wait_until parameter for checking whether the tag with the name html has appeared on the web page or not, it will keep on checking until it appears on the web page.
The don’t_filter set as True will allow the revisiting of the website with the same domain name.
Step 5: Extracting emails from the main page of each website 
Python3
# extracting emailsdef parse(self, response):    EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'    emails = re.finditer(EMAIL_REGEX, str(response.text))    for email in emails:        self.email_list.append(email.group())     for email in set(self.email_list):        yield{            "emails": email        }     self.email_list.clear()
  In this step we are creating a method called parse() with the parameter response for getting request object from the start_request method, in the next line we are creating our regular expression system for parsing out the emails from the response HTML then we are appending the emails in the email_list list variable which we declared in the constructor method, and then we are iterating over the set and yielding a dictionary where emails is key or column header and email is an iterator or relative email value and at the very last we are clearing the list so that no duplicate values are written to the file when we start the crawler.
Below is the complete program based on the above approach: 
Python3
# import required modulesimport scrapyfrom scrapy.spiders import CrawlSpider, Requestfrom googlesearch import searchimport refrom scrapy_selenium import SeleniumRequestfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as EC  # create class to extract email idsclass email_extractor(CrawlSpider):         # adjusting parameters    name = 'email_ex'     def __init__(self, *args, **kwargs):        super(email_extractor, self).__init__(*args, **kwargs)        self.email_list = []        self.query = " 'market places'.gmail.com "     # sending requests    def start_requests(self):        for results in search(self.query, num=10, stop=None, pause=2):            yield SeleniumRequest(                url=results,                callback=self.parse,                wait_until=EC.presence_of_element_located(                    (By.TAG_NAME, "html")),                dont_filter=True            )     # extracting emails    def parse(self, response):        EMAIL_REGEX = r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+'        emails = re.finditer(EMAIL_REGEX, str(response.text))        for email in emails:            self.email_list.append(email.group())         for email in set(self.email_list):            yield{                "emails": email            }         self.email_list.clear()
Now it’s time to run the code, open the terminal and go to the root directory of the project where scrapy.cfg file is located and run this command:
scrapy crawl email_ex -o emails.csv
Scraper will start scraping and storing all the emails to the file emails.csv that is created automatically.
And so the results are:
Extracted emails are stored in a csv file 
ahmadwaqar
Python web-scraping-exercises
Python-projects
Python-selenium
python-utility
Technical Scripter 2020
Web-scraping
Python
Technical Scripter
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python Classes and Objects
How to drop one or multiple columns in Pandas Dataframe
Defaultdict in Python
Python | Get unique values from a list
Python | os.path.join() method
Create a directory in Python
Python | Pandas dataframe.groupby()