Package: tinyCrawl
Overview 概览tinyCrawl 是一个微型的爬虫框架，具有以下特点：简单轻巧，没有任何第三方包的依赖checkpoint断点续爬支持多线程爬取内置日志功能使用简单Documentation 文档访问官方说明文档Installation 安装pip install tinyCrawlHow to use 如何使用tinyCrawl 支持2种运行方法By using funciton 函数式：定义爬虫的方法 task()，实例化BaseCrawl(iter_url, iter_num_range, thread_num)，调用run方法执行By inheritance 继承式：继承BaseCrawl(iter_url, iter_num_range, thread_num)，重写crawl()和sink()方法，其中crawl()类似于上一方法中的爬虫方法task()，定义爬取单页的爬虫代码，sink()是将结果输出的方法，最后执行main()方法执行总程序By using funciton 函数式# -*- coding: utf-8 -*-fromtinyCrawlimportBaseCrawl,RowContainerfromurllib.requestimporturlopenfromlxmlimportetree# 定义xpathsong_name_xpath='//div[@class="song-name"]/a/text()'singer_xpath='//div[@class="singers"]/a[1]/text()'album_xpath='//div[@class="album"]/a[1]/text()'deftask(url):"""定义爬取单页的爬虫代码"""# 定义数据存放的容器，容器名字就是最后爬取结果存放字典的self.out的keysong_name_list=RowContainer("song name")singer_list=RowContainer("singer")album_list=RowContainer("album")page=urlopen(url).read().decode("utf-8",'ignore')parse=etree.HTML(page)for_song_name,_singer,_albuminzip(parse.xpath(song_name_xpath),parse.xpath(singer_xpath),parse.xpath(album_xpath)):# 将数据append进指定容器中song_name_list.append(str(_song_name))singer_list.append(str(_singer))album_list.append(str(_album))# 第一个参数是链接地址，需通过 %s 定义页数等迭代的参数# 第二个参数是迭代的范围，# 第三个参数是启用的线程数，大于1就是多线程，等于1就是单线程bc=BaseCrawl("http://example.com/?page=%s",range(1,5),3)# 输入task的对象，开始执行程序bc.run(task)# 执行完毕后，通过out属性，获取结果print(bc.out)By inheritance 继承式fromtinyCrawlimportBaseCrawl,RowContainerfromurllib.requestimporturlopenfromlxmlimportetreeimportpandasaspd# 需继承BaseCrawl类，覆写crawl和sink方法classScratch(BaseCrawl):def__init__(self,iter_url,iter_num_range,thread_num):super().__init__(iter_url,iter_num_range,thread_num)# 覆写crawl方法defcrawl(self,url):# 定义数据存放的容器，容器名字就是最后爬取结果存放字典的self.out的keysong_name_list=RowContainer("song name")singer_list=RowContainer("singer")album_list=RowContainer("album")page=urlopen(url).read().decode("utf-8",'ignore')parse=etree.HTML(page)for_song_name,_singer,_albuminzip(parse.xpath(song_name_xpath),parse.xpath(singer_xpath),parse.xpath(album_xpath)):# 将数据append进指定容器中song_name_list.append(str(_song_name))singer_list.append(str(_singer))album_list.append(str(_album))# 覆写sink方法，将爬取的结果输出defsink(self):# self.out是字典结构的结果，可以直接输入pandas存为dataframerecent_music=pd.DataFrame(self.out)recent_music.to_csv("D:/tmptest.csv",index=0)if__name__=='__main__':mc=Scratch("http://example.com/?page=%s",range(1,5),3)# 调用main函数执行程序mc.main()output：2021-01-1016:18:36,944-base.py-__init__-[line:30]-INFO:Checkpointpath:D:\breakpoint_page.txt2021-01-1016:18:38,539-base.py-__source-[line:119]-INFO:Nowisrunningonmultithreadmode,totalthreadnumis`3`2021-01-1016:18:38,539-base.py-__source-[line:126]-INFO:Totaliterationnum:42021-01-1016:18:38,541-base.py-_multi_thread_wrap-[line:59]-INFO:ThreadPoolExecutor-1_0nowisprocessing:http://example.com/?page=12021-01-1016:18:38,541-base.py-_multi_thread_wrap-[line:59]-INFO:ThreadPoolExecutor-1_1nowisprocessing:http://example.com/?page=22021-01-1016:18:38,542-base.py-_multi_thread_wrap-[line:59]-INFO:ThreadPoolExecutor-1_2nowisprocessing:http://example.com/?page=32021-01-1016:18:41,544-base.py-__task_done-[line:115]-INFO:ThreadPoolExecutor-1_1taskfinished;(Timetook:3.0009s)2021-01-1016:18:41,544-base.py-__task_done-[line:115]-INFO:ThreadPoolExecutor-1_0taskfinished;(Timetook:3.0019s)2021-01-1016:18:41,544-base.py-__task_done-[line:115]-INFO:ThreadPoolExecutor-1_2taskfinished;(Timetook:3.0009s)2021-01-1016:18:41,545-base.py-_multi_thread_wrap-[line:59]-INFO:ThreadPoolExecutor-1_1nowisprocessing:http://example.com/?page=42021-01-1016:18:44,551-base.py-__task_done-[line:115]-INFO:ThreadPoolExecutor-1_1taskfinished;(Timetook:3.0022s)2021-01-1016:18:44,551-base.py-__source-[line:151]-INFO:Alldone.(Timetook:6.0102s)