Saving scraped items to JSON and CSV file using Scrapy - GeeksforGeeks

09 Aug, 2021
In this article, we will see how to use crawling with Scrapy, and, Exporting data to JSON and CSV format. We will scrape data from a webpage, using a Scrapy spider, and export the same to two different file formats.
Here we will extract from the link  http://quotes.toscrape.com/tag/friendship/. This website is provided by the makers of Scrapy, for learning about the library. Let us understand the above approach stepwise:
Step 1: Create scrapy project
Execute the following command, at the terminal, to create a Scrapy project –
scrapy startproject gfg_friendshipquotes 
This will create a new directory, called “gfg_friendshipquotes”, in your current directory. Now change the directory, to the newly created folder. 
The folder structure of ‘gfg_friendshipquotes’ is as displayed below.  Keep the contents of the configuration files as they are, currently.
Step 2: To create a spider file, we use the command ‘genspider ‘. Please see that genspider command is executed at the same directory level, where scrapy.cfg file is present. The command is –
scrapy genspider spider_filename “url_of_page_to_scrape” 
Now, execute the following at the terminal:
scrapy genspider gfg_friendquotes “quotes.toscrape.com/tag/friendship/”
This should be created, a spider Python file, called “gfg_friendquotes.py”, in the spiders folder as:
The default code of the gfg_friendquotes.py file is as follows:
Python
# Import the required libraryimport scrapy # Spider classclass GfgFriendquotesSpider(scrapy.Spider):      # The name of the spider    name = 'gfg_friendquotes'         # The domain, the spider will crawl    allowed_domains = ['quotes.toscrape.com/tag/friendship/']         # The URL of the webpage, data from which    # will get scraped    start_urls = ['http://quotes.toscrape.com/tag/friendship/']         # default start function which will hold    # the code for navigating and gathering    # the data from tags    def parse(self, response):        pass
Step 3: Now, let’s analyze the XPath expressions for required elements. If you visit the link, http://quotes.toscrape.com/tag/friendship/ it looks as follows:
URL of the page that we will scrape
We are going to scrape the friendship quotes titles, authors, and tags. When you right-click on Quotes, block it, and select Inspect option, one can notice they belong to class “quote”. As you hover over the rest of the quote blocks, one can notice that all the quotes, in the webpage, have the CSS class attribute as “quote”.
Right-Click, Inspect, check CSS attributes of first Quote block
To extract, the text of the quote, right-click on the first quote, and, say Inspect. The title/text of the quote, belongs to the CSS class attribute, “text”.
Right-Click first Title, Inspect, check CSS class attributes 
To extract the author name of the quote, right-click on the first name, and, say Inspect. It belongs to the CSS class “author”. There is an itemprop CSS attribute, defined here as well with the same name. We will use this attribute in our code.
 Right-Click on author name to get its CSS attributes
Step 7: To extract the tags of the quote, right-click on the first tag, and, say Inspect. A single tag belongs to the CSS class “tag”. Together, they have an itemprop CSS attribute, “keywords” defined. They also have a “content” CSS attribute, with all the tags in one line. If you observe, the actual text of tags is present inside <a>, hyperlink elements. Hence, fetching from the “content” attribute would be easier. 
 Right-Click on Tags  to get its CSS attributes
The final code, for the spider file, after including the XPath expressions, is as follows –
Python3
# Import the required librariesimport scrapy # Default class created when we run the "genspider" command  class GfgFriendquotesSpider(scrapy.Spider):    # Name of the spider as mentioned in the "genspider" command    name = 'gfg_friendquotes'    # Domains allowed for scraping, as mentioned in the "genspider" command    allowed_domains = ['quotes.toscrape.com/tag/friendship/']    # URL(s) to scrape as mentioned in the "genspider" command    # The scrapy spider, starts making  requests, to URLs mentioned here    start_urls = ['http://quotes.toscrape.com/tag/friendship/']     # Default callback method responsible for returning the scraped output and processing it.    def parse(self, response):       # XPath expression of all the Quote elements.        # All quotes belong to CSS attribute class having value 'quote'        quotes = response.xpath('//*[@class="quote"]')        # Loop through the quotes object, to get required elements data.        for quote in quotes:            # XPath expression to fetch 'title' of the Quote            # Title belong to CSS attribute class having value 'text'            title = quote.xpath('.//*[@class="text"]/text()').extract_first()            # XPath expression to fetch 'author name' of the Quote            # Author name belong to CSS attribute itemprop having value 'author'            author = quote.xpath(                './/*[@itemprop="author"]/text()').extract_first()            # XPath expression to fetch 'tags' of the Quote            # Tags belong to CSS attribute itemprop having value 'keywords'            tags = quote.xpath(                './/*[@itemprop="keywords"]/@content').extract_first()            # Return the output            yield {'Text': title,                   'Author': author,                   'Tags': tags}
Scrapy allows the extracted data to be stored in formats like JSON, CSV, XML etc. This tutorial shows two methods of doing so. One can write the following command at the terminal:
scrapy crawl “spider_name” -o store_data_extracted_filename.file_extension
Alternatively, one can export the output to a file, by mentioning FEED_FORMAT and FEED_URI in the settings.py file. 
For storing the data in a JSON file, one can follow any of the methods mentioned below:
scrapy crawl gfg_friendquotes -o friendshipquotes.json
Alternatively, we can mention FEED_FORMAT and FEED_URI in the settings.py file. The settings.py file should be as follows:
Python
BOT_NAME = 'gfg_friendshipquotes' SPIDER_MODULES = ['gfg_friendshipquotes.spiders']NEWSPIDER_MODULE = 'gfg_friendshipquotes.spiders' # Obey robots.txt rulesROBOTSTXT_OBEY = True # Desired file formatFEED_FORMAT = "json" # Name of the file where# data extracted is storedFEED_URI = "friendshipfeed.json"
 Output:
Using any of the methods above, the  JSON files are generated in the project folder as:
The extracted data, exported to JSON  files
The expected JSON file looks as follows: 
The Exported JSON data, crawled by spider code
For storing the data in a CSV file, one can follow any of the methods mentioned below.
Write the following command at the terminal: 
scrapy crawl gfg_friendquotes -o friendshipquotes.csv
Alternatively, we can mention FEED_FORMAT and FEED_URI in the settings.py file. The settings.py file should be as follows:
Python
BOT_NAME = 'gfg_friendshipquotes' SPIDER_MODULES = ['gfg_friendshipquotes.spiders']NEWSPIDER_MODULE = 'gfg_friendshipquotes.spiders' # Obey robots.txt rulesROBOTSTXT_OBEY = True # Desired file formatFEED_FORMAT = "csv" # Name of the file where data extracted is storedFEED_URI = "friendshipfeed.csv"
Output:
The CSV files are generated in the project folder as:
The exported files are created in your scrapy project structure
The exported CSV file looks as follows:
The Exported CSV data, crawled by spider code
phadnispradnya
Picked
Python-Scrapy
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Comments
Old Comments
How to Install PIP on Windows ?
How to drop one or multiple columns in Pandas Dataframe
How To Convert Python Dictionary To JSON?
Check if element exists in list in Python
Python | Pandas dataframe.groupby()
Defaultdict in Python
Python | Get unique values from a list
Python Classes and Objects
Python | os.path.join() method
Create a directory in Python