How to Collect Product and Supplier Data from Alibaba with Using python?
By: M Alaaddin Çelik
Tags: Suppliers, Web Scraping, Python, Alibaba

In this article, there is an explanation of a python class which is to be used to collect data from alibaba.com product page. The name of the python class is Alibaba whose source code can be found on github. The goal is to gather some kind of data which can be used in the market research. There are two kinds of data category. One of them is related to the product data such as name, minimum order and price information. The other is about supplier information which are name, link, country, category and response rate.

The requirements to understand the text are to know python, object-oriented programming and some html syntax. Additionally, requests and Beautiful-Soap libraries will be used to scrape and extract data. For that reason, python3, requests and Beautiful-Soap must be installed in your system.

The main concept is to collect data with using alibaba class but there will be an explanation about how to use requests and Beautiful-Soap libraries to collect data from the product page.

How to Use requests and Beautiful-Soap Libraries

Firstly, we have to import libraries as it can be seen in the figure 1.

figure 1

Then, we can choose the product to be scrape. In that case, The product to be scraped will be a smart bracelet. The link of the product can be assigned as a variable like in figure 2.

figure 2

Now, we can send a request to the page for taking a response. In figure 3, there is a variable as a response which represents the status from the web page. To understand the response status, the print function can be used. If the result is “<Response [200]>” , it means we can continue to our code.

figure 3

After having successful response, we can make our soup variable as in figure 4. This variable represents the html structure of the page. It will be used to parse the html structure so that we can find and extract our desired data.

figure 4

Finally, we can find the our desired data ,in that case, it is the name of the product. When we look into the html structure like in figure 5, there are some information about the name data. For example, it is written with a h1 html tag, the class name is ‘ma-title’. The developer tool can be used to analyze the html syntax more deeply. Then, the information about the data can be used like in figure 6.

figure 5

figure 6

the last version of the code

How to Use Alibaba Class to Collect Data?

In the above section, the usage of requests and Beautiful-soap libraries have been explained. These libraries are used to create Alibaba class. In this section, the same product, a smart bracelet, will be scraped with using Alibaba class. In the end, there will be a json file with some product and supplier data.

Let’s begin with importing the necessary classes.

Then , we need to assign the product link like in the above section.

Now, we are ready to make an instance to Alibaba class. Let’s give a name the instance as product_page.

As it mentioned before, there are two kinds of data category. In figure 7, two methods are used to collect the product data. The product_price_data represents the relationship the between quantities and prices.

figure 7

In figure 8, four methods are used to gather supplier data.

figure 8

At this moment, there are variables to hold the desired data but we need to add them into the dictionary. As it is shown in figure 9, there is a dictionary called data.

figure 9

Finally, we can export our data from the workstation. The code in figure 10 will create a json file with name product_page_data.

figure 10

json format of data