Python scrapy tutorial for beginners — 02 — Extract all the data!
Tags: Data Mining, Programming, Python, Web Development, Scrapy

Extracting time — Different ways to pull data

As we did on the parse method, we are going to extract the data from each own book URL. Open one random book, for example, Sharp Objects

We are going to use this one as a model and every book will be scraped the same way.

We have a lot to choose from! Why don’t we start from the title?

Extracting data — The easy ones

Right-click on the title, select inspect and look where it is located. It’s just the only h1 tag after a div. Pretty easy. Let’s find one h1 after a div, and extract the text. Then, we store it in a variable:

Let’s run the code and print the title:

Easy, right?

Before, we just had the main URL and loop over the articles to extract the data.

Now we have the main URL and loop over the articles to extract the URL, then request the new URL and we extract the data. One additional step in another method. This is all it takes.

Let’s keep going. Locate the image and right-click it and then inspect it. Seems like we have a partial URL again!

Luckily you have learnt a lot in our first lesson and you know how to create the final URL by getting the partial URL and adding the base URL. Why don’t you give it a try?

Doesn’t matter if you don’t succeed at the first try. Get the URL, add the base URL and print the result until you find it.

This is how I did it:

As always, print final_image to see that you have a proper URL. You know the drill.

Let”s get the price.

The ‘contains’ selector

Right-click the price, inspect it and you can see that it is inside a p tag with a price_color class.

The problem is that every item at the bottom section of ‘Products you recently viewed’ have that too!

We not only need to search for the price searching for a p tag with the price_color class inside a div, that div also need to have a product_main class!

But that is just one part of the class:

We can use a selector to search for an item that its class contains a string. Instead of using the whole class, “col-sm-6 product_main”, we are only search for product_main.

Here’s the code:

price = response.xpath( '//div[contains(@class, "product_main")]/p[@class="price_color"]/text()').extract_first()

We look for a div that its class contains product_main, then we get the text inside the p with price_color class.

Print the price and run the code again to check it is working.

Now, your turn: Scrape the stock (The text that says ‘ In stock (X available) ‘). Use the technique you just have seen and do it yourself.

Here’s my solution:

stock = response.xpath( '//div[contains(@class, "product_main")]/p[contains(@class, "instock")]/text()').extract()[1].strip()

This time we have 2 elements, so I extract the desired and I remove the empty spaces with python .strip().

Let’s extract the ratings. Right-click on the stars and we have this:

Every star has a icon-star class, but if you watch the previous div, you can see that all the stars are wrapped around in div with star-rating Four class. Four is the rating.

Try to extract it. Just get a p that contains the star-rating class and get that class. Remove the extra text we don’t need.

Here’s my code:

stars = response.xpath( '//div/p[contains(@class, "star-rating")]/@class').extract_first().replace('star-rating ', '')

Family matters — Siblings

The description is a tricky one:

The p tag has no class! How can we select it?

Well, we can’t… But we can select the previous element, div id=”product_description”, then select the next HTML node, or it sibling. Like this:

description = response.xpath( '//div[@id="product_description"]/following-sibling::p/text()').extract_first()

We select the div with the id product_description, then we go to the next p sibling and we select and extract the text. Phew!

Tables

As if you didn’t had enough with contains and siblings, now we have tables!

Don’t you worry, I have you covered.

We need to select the table, the row or tr, then the position of said row, and then the value, in this case, td. After the selection, we get the text as usual. Let me do the first one, UPC:

upc = response.xpath( '//table[@class="table table-striped"]/tr[1]/td/text()').extract_first()

Print it and run the spider. This is how we extract data from tables. Now it’s your turn:

Extract the price excl tax, price inc tax and tax. As we did on the first spider, yield the result as we did on the first spider.

Do it yourself and don’t look here unless needed.

And that’s it! Run the spider but this time, store the file into a file.

scrapy crawl spider -o books_detailed.json

Open the new file and make sure everything is in order.