Network data analysis of crawler general search engine based on Python
Li Yapeng(College of Urban and Environmental Sciences, Liaoning Normal University, Dalian 116029, China)

Abstract
With the development of information age and the popularization of programming technology,
search engine has become a necessity in people’s daily life. Most search engines use crawler
technology as the core module to return the results of user queries through keywords. However, theexplosive growth of network information makes it difficult to locate and locate information. In viewof the above problems, this paper takes the general search engine “Baidu Search” as the crawling
object on the basis of Python. On the basis of learning and analyzing the principle, core modules andrunning process of current crawling technology, this paper compares three different data crawling
methods to achieve the goal of data crawling. Firstly, this paper gives the principle and workflow ofcrawler technology, introduces some key technologies in crawler engineering, and focuses on themethod of crawling web pages. It is intended to provide reference for future research on the problems
and the possibility of improvement.
Key word: General search engine, Python crawler, crawl web■ 1. General Search Enging work principle
Web crawler can be divided into two kinds:General crawler and focused crawler. General web
crawler is to collect web pages and informationfrom the Internet[1~3]. These web pages are used toprovide support for the search engine indexing[4,5].
It determines whether the content of the wholeengine system is rich and the information istimely. Therefore, the performance of the crawler
directly affects the effect of the search engine.
Focused crawler is a kind of web crawler programfor specific subject requirements. It differs fromgeneral search engine crawler in that the focused
crawler processes and filters the content when it
implements web crawling, so as to ensure that
only web pages related to the requirements arecrawled.

26

NAIS Journal

1.1 Crawling
1.1.1 Basic workflow
The first part selects the seed URLs, which arecaptured in the URL queue[6]; removed to the crawl
URL, DNS analyzes the host IP, URLs, and thecorresponding page downloads[7]; the download
pages are stored in the database, and these pages
have been placed in the URL queue crawl URL.
Analyze the URL in the crawled URL queue,
analyze other URLs, and put the URL into thequeue to be crawled, leading to the next loop
(Fig.1).

1.3 Preprocessing
The search engine takes the crawler’s pages
back and preprocessed various steps. Extract text,
Chinese segmentation, noise elimination (such as
copyright text, navigation, advertising...), index ofprocessing, link calculation, special document
processing etc.. In addition to HTML files, searchengines can also capture and index text-based filetypes such as PDF, WORD, WPS, PPT, TXT, etc.
We often see this type of file in search results. Butsearch engines can’t handle non-verbal content
such as pictures, videos, Flash, or execute scripts
Fig. 1 Workflow diagram

1.1.2 Get URL
The new site offers sites to search engines on its own
initiative: (http://yapengli.baidu.com/linksubmit/url),and programs.

■ 2. Page analysis

Extraction from the web crawler to grab somelinks to new sites on other sites (as far as possible

data to achieve some purpose. A method ofwithin the scope of search engine crawling), search

extracting web data: Beautiful, Soup and lxml

engines and DNS resolution services (such as DNSPod)

regular expressions. With the option through thecooperate, and new site domain names will be quickly

browser, view the page source code, through thecrawled. But search engine spider crawling is typed

Firebug Lite extension (http://getfirebug.com/

with certain rules, it needs to comply with some

firebuglite), analysis of web information. Firefox

commands or file content, such as the link labeledcan install the full version of the Firebug browser.

nofollower, or the Robots protocol. Robots protocol (alsoknown as the crawler protocol, robot protocol, etc.), the

2.1 Three scraping method

full name is “Robots Exclusion Protocol” (Robots

2.1.1 Regular expressions

Exclusion Protocol), the site through the Robots

Python regular expression (2.x): https://docs.

protocol tells search engines which pages can bepython.org/2/howto/regex.html can grab data bycrawled, which pages can not be crawled, such as:matching a single page element, but regularTaobao: https://www.taobao.com/robots.txt; Tencent:expressions often fail if the page changes. A more

http://ww. W.qq.com/robots.txt.

robust way is to add the parent element of the1.2 Data storage

unique identifier of the target web page to theSearch engines crawl through the crawler to thematching rule[8].

web page and store the data in the original page

import urllib2

database. The page data and the user browser

import rewhich is the same as that of the obtained HTML.

def scrape(html):

Search engine spiders also do a certain amount ofarea = re.findall(‘<tr id=”places_area__

duplicate content detection when crawling pages.

row”>.*?<td\s*class=[“\’]w2p_fw[“\’]>(.*?)</

Once encountering a lot of plagiarized, collected ortd>’, html)[0]

duplicated content on sites with very low accessreturn area

weight, it is likely that they will not crawl anymore.

if __name__ == ‘__main__’:html=

urllib2.urlopen(‘http://example.

webscraping. com/view/United-Kingdom-239’).read()NAIS Journal27print scrape(html)

2.1.3 Lxml

Generally speaking, the method of regular

Lxml is based on the Python encapsulation ofexpression is not suitable for the scene of frequently

libxml2 XML analytic library. Http://lxml.de/,changing web pages, and it has some problems

http://lxml.de/installation.html. The CSS selector

such as difficult to construct and poor readability.

represents the mode used by the selection element.

2.1.2 Beautiful Soup

Compared with the XPath selector, the CSS

Beautiful Soup is a Python library that canselector is more succinct. But in the internal

extract data from HTML or XML files: https://www.

implementation of Lxml, the CSS selector is

crummy.com/software/Beautiful Soup/. Compared

actually converted to an equivalent XPathwith regular expressions, code using Beautiful

selector[10].

Soup is easier to construct and understand.

# -*- coding: utf-8 -*-Installation module: PIP install beautifullysoup4-i

import urllib2

https://mirrors.ustc.edu.cn/pypi/web/simple/Using

import lxml.html

Beautiful Soup, you first parse the downloaded

def scrape(html):

HTML content into a source document to determine

tree = lxml.html.fromstring(html)

the actual format; then use find () and find_all () to

td = tree.cssselect(‘tr#places_area__row >locate the required elements .

td.w2p_fw’)[0][9]# -*- coding: utf-8 -*-area = td.text_content()

import urllib2

return areafrom bs4 import BeautifulSoup

if __name__ == ‘__main__’:def scrape(html):

html

=

urllib2.urlopen(‘http://example.

soup = BeautifulSoup(html, “html.parser”)

webscraping.com/view/United-Kingdom-239’).

tr = soup.find(attrs={‘id’: ‘places_area__read()row’}) # locate the area row

print scrape(html)# ‘class’ is a special python attribute soinstead ‘class_’ is used

2.2 Comparison and analysis

td = tr.find(attrs={‘class’: ‘w2p_fw’}) #locate the area tag

usually the best choice for capturing data, whilearea = td.text # extract the area contentsfrom this tag
if __name__ == ‘__main__’:=regular expressions and Beautiful Soup are useful
only in certain scenarios(Table.1).

return areahtml

Lxml methods are fast and robust and are

urllib2.urlopen(‘http://example.

■ 3. Conclusions

The amount of information on the web was

webscraping.com/view/United-Kingdom-239’).

“explosive” growth today, users need from the vastread()

amounts of information can accurately extract therequired information technology. In this context,

print scrape(html)

Table 1 Comparison and analysis of three Python Crawlers

28

NAIS Journal

Grab method

Property

Facility value

Installation

Regular expressions

fast

difficulty

Simple (built-in module)

Beautiful Soupslow

simpleSimple (pure Python)Lxml

fastsimple

relative difficultyWeb crawler technology will continue to attract
people’s attention because of its powerful ability toautomatically extract web information.
Reference[1] Guo Erqiang, Li Bo. Web crawler technology based on
Python in large data environment [J].Computer
products and circulation, 2017 (12): 82.
[2 Bird S, Klein E, Loper E. Natural Language
Processing with Python[M]. Southeast University
press, 2010.
[3 Manning C D, Surdeanu M, Bauer J, et al. The
Stanford CoreNLP Natural Language Processing
To o l k i t [ C ] / / M e e t i n g o f t h e A s s o c i a t i o n f o rComputational Linguistics: System Demonstrations.
2014.
[4] Zhou Zhonghua, Zhang Huiran, Xie Jiang.Pythonbased data crawler on Sina Weibo [J].ComputerApplications, 2014, 34 (11): 3131-3134.
[5] Lianjie, Zhou Xin, Cao Wei, et al. Data Mining
Scheme of Sina Weibo [J].Journal of Tsinghua
University: Natural Science Edition, 2011, 51 (10):1300-1305.
[6] Chen Meng.Design and Implementation of a Pythonbased Sina News Reptile System[J].Modern
Information Technology, 2018,2(07): 111-112.

[7] Pan Qiaozhi, Zhang Lei.A Brief Introduction toPython-based Network Crawler Technology in Large
Data Environment [J].Network Security Technologyand Applications, 2018 (05): 41-42.
[8] Bai Xueli. Analysis of the characteristics andapplication of Python based reptile technology [J].
Shanxi science and technology, 2018,33 (02): 53-55.
[9] Xiong Chang.Web page data capture and analysis
based on Python crawler technology[J].Digitaltechnology and applications, 2017 (09): 35-36.
[10] Fang Jin Tang. Design and implementation of online
education platform based on web crawler [D]. Beijing
Jiaotong University, 2016.

◆著者紹介

Li Yapeng「Master of Quaternary Geology, school of Urban andEnvironmental Sciences, Liaoning Normal University,Dalian,China

NAIS Journal

29