How to Run Scrapy From a Script. Forget about scrapy’s framework and... | by Aaron S | Towards Data Science

Scrapy is a great framework to use for scraping projects. However, did you know there is a way to run Scrapy straight from a script?
Looking at the documentation, there are two ways to run Scrapy. Using the Scrapy API or the framework.
Why you would use scrapy from a scriptUnderstand the basic script every time you want access scrapy from an individual scriptUnderstand how to specify customised scrapy settingsUnderstand how to specify HTTP requests for scrapy to invokeUnderstand how to process those HTTP responses using scrapy under one script.
Why you would use scrapy from a script
Understand the basic script every time you want access scrapy from an individual script
Understand how to specify customised scrapy settings
Understand how to specify HTTP requests for scrapy to invoke
Understand how to process those HTTP responses using scrapy under one script.
Scrapy can be used for a heavy-duty scraping work, however, there are a lot of projects that are quite small and don’t require the need for using the whole scrapy framework. This is where using scrapy in a python script comes in. No need to use the whole framework you can do it all from a python script.
Let’s see what the basics of this look like before fleshing out some of the necessary settings to scrape.
The key to running scrapy in a python script is the CrawlerProcess class. This is a class of the Crawler module. It provides the engine to run scrapy within a python script. Within the CrawlerProcess class code, python’s twisted framework is imported.
Twisted is a python framework that is used for input and output processes like HTTP requests for example. Now it does this through what’s called a twister event reactor. Scrapy is built on top of twisted! We won’t go into too much detail here but needless to say, the CrawlerProcess class imports a twisted reactor which listens for events like multiple HTTP requests. This is at the heart of how scrapy works.
CrawlerProcess assumes that a twisted reactor is NOT used by anything else, like for example another spider. With that, we have the code below.
import scrapyfrom scrapy.crawler import CrawlerProcessclass TestSpider(scrapy.Spider):    name = 'test'if __name__ == "__main__":  process = CrawlerProcess()  process.crawl(TestSpider)  process.start()
Now for us to use the scrapy framework, we must create our spider, this is done by creating a class which inherits from scrapy.Spider. scrapy.Spider is the most basic spider that we must derive from in all scrapy projects. With this, we have to give this spider a name for it to run/ Spiders will require a couple of functions and an URL to scrape but for this example, we will omit this for the moment.Now you see if __name__ == “__main__”. This is used as a best practice in python. When we write a script you want to it to be able to run the code but also be able to import that code somewhere else. Please see here for further discussion on this point.We instantiate the class CrawlerProcess first to get access to the functions we want. CrawlerProcess has two functions we are interested in, crawl and startWe use crawl to start the spider we created. We then use the start function to start a twisted reactor, the engine that processes and listens to our HTTP requests we want.
Now for us to use the scrapy framework, we must create our spider, this is done by creating a class which inherits from scrapy.Spider. scrapy.Spider is the most basic spider that we must derive from in all scrapy projects. With this, we have to give this spider a name for it to run/ Spiders will require a couple of functions and an URL to scrape but for this example, we will omit this for the moment.
Now you see if __name__ == “__main__”. This is used as a best practice in python. When we write a script you want to it to be able to run the code but also be able to import that code somewhere else. Please see here for further discussion on this point.
We instantiate the class CrawlerProcess first to get access to the functions we want. CrawlerProcess has two functions we are interested in, crawl and start
We use crawl to start the spider we created. We then use the start function to start a twisted reactor, the engine that processes and listens to our HTTP requests we want.
The scrapy framework provides a list of settings that it will use automatically, however for working with the Scrapy API we have to provide the settings explicitly. The settings we define is how we can customise our spiders. The spider.Spider class has a variable called custom_settings. Now this variable can be used to override the settings scrapy automatically uses. We have to create a dictionary of our settings to do this as the custom_settings variable is set to none using scrapy.
You may want to use some or most of the settings scrapy provides, in which case you could copy them from there. Alternatively, a list of the built-in settings can be found here.
class TestSpider(scrapy.Spider):    name = 'test'    custom_settings = { 'DOWNLOD_DELAY': 1 }
We have shown how to create a spider and define the settings, but we haven’t specified any URLs to scrape, or how we want to specify the requests to the website we want to get data from. For example, parameters, headers and user-agents.When we create spider we also start a method called start_requests(). This will create the requests for any URL we want. Now there are two ways to use this method.
1) By defining the start_urls attribute 2) We implement our function called start_requests
The shortest way is by defining start_urls. We define it as a list of URLs we want to get, by specifying this variable we automatically use start_requests() to go through each one of our URLs.
class TestSpider(scrapy.Spider):    name = 'test'    custom_settings = { 'DOWNLOD_DELAY': 1 }    start_urls = ['URL1','URL2']
However notice how if we do this, we can’t specify our headers, parameters or anything else we want to go along with the request? This is where implementing our start_requests method comes in.
First, we define our variables we want to go along with the request. We then implement our start_requests method so we can make use of the headers and parameters we want, as well as where we want to the response to go.
class TestSpider(scrapy.Spider):    name = 'test'    custom_settings = { 'DOWNLOD_DELAY': 1 }    headers = {}     params = {}    def start_requests(self):        yield scrapy.Requests(url, headers=headers, params=params)
Here we access the Requests method which when given an URL will make the HTTP requests and return a response defined as the response variable.
You will notice how we didn’t specify a callback? That is we didn’t specify where scrapy should send the response to the requests we just told it to get for us.
Let’s fix that, by default scrapy expects the callback method to be the parse function but it could be anything we want it to be.
class TestSpider(scrapy.Spider):    name = 'test'    custom_settings = { 'DOWNLOD_DELAY': 1 }    headers = {}     params = {}    def start_requests(self):        yield scrapy.Requests(url, headers=headers, params=params,callback = self.parse)   def parse(self,response):       print(response.body)
Here we have defined the function parse which accepts a response variable, remember this is created when we ask scrapy to do the HTTP requests. We then ask scrapy to print the response body.
With that, we now have the basics of running scrapy in a python script. We can use all the same methods but we just have to do a bit of configuring beforehand.
Why might you use the scrapy framework ? When is importing scrapy in a python script useful?What does the CrawlerProcess class do?Can you recall the basic script used to start scrapy within a python script?How do you add scrapy settings in your python script?Why might you use a start_requests function instead of start_urls ?
Why might you use the scrapy framework ? When is importing scrapy in a python script useful?
What does the CrawlerProcess class do?
Can you recall the basic script used to start scrapy within a python script?
How do you add scrapy settings in your python script?
Why might you use a start_requests function instead of start_urls ?
Please see here for further details about what I’m up to project-wise on my blog and other posts.For more tech/coding related content please sign up to my newsletter here
I’d be grateful for any comments or if you want to collaborate or need help with python please do get in touch.If you want to get in contact with me, please do so here asmith53@ed.ac.uk.