Package: com.dvsnier.sniffer
Python NetWork Sniffer一. 配置1.1. 脚本配置1.1.1. 嗅探单页1.1.2. 嗅探多页1.2. 文件配置1.3. CLI 配置二. 运行2.1. 脚本运行2.1. CLI 运行一. 配置1.1. 脚本配置打开.\scripts\crawler_script.py脚本配置如下:1.1.1. 嗅探单页嗅探单页数据如下所示:if__name__=="__main__":'''主函数入口'''page_size=1crawler=Crawler_Bbs_TianYa()# crawler.set_flag(False)crawler.set_range(page_size).run()1.1.2. 嗅探多页嗅探[1, 5)页数据如下所示:方式一:if__name__=="__main__":'''主函数入口'''page_size=5crawler=Crawler_Bbs_TianYa()# crawler.set_flag(False)crawler.set_range(page_size).run()方式二:if__name__=="__main__":'''主函数入口'''crawler=Crawler_Bbs_TianYa()# crawler.set_flag(False)crawler.set_range(page_start=1,page_stop=5).run()1.2. 文件配置文件配置列表如下:# the version informationversion_name=v0.0.1.dev1version_code=1version_info=0.0.1.dev1# CRAWLER URL PREFIXarticle-alias='crawler_alias'sn-url-prefix=['http://xxx.yyy.zzz/post-xxx-yyy-{}.shtml']# REGION_INCLUSIVE_EXCLUSIVE = 0# REGION_EXCLUSIVE_INCLUSIVE = 1# REGION_EXCLUSIVE_EXCLUSIVE = 2# REGION_INCLUSIVE_INCLUSIVE = 3page-start=1page-stop=0page-flag=0# page-flag = 1# page-flag = 2# page-flag = 3# False: first pull , second translate True: one pull after another translatearticle-flag=True# article-flag = False# True: multi media resources are stored locally, otherwise they are not# article-multi-media-persistence = True# article-multi-media-persistence = False# True: multi media resources are high quality, otherwise they are not# article-multi-media-quality = True# article-multi-media-quality = FalseUser-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/512.36 (KHTML, like Gecko) Chrome/92.0.1235.131 Safari/277.36'# User-Agent = 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_10_3) AppleWebKit/512.36 (KHTML, like Gecko) Chrome/92.0.1235.131 Safari/277.36'# output-directory = "..."# output-uuid-encryptor = True# output-uuid-encryptor = False说明如下:version_name: 配置文件版本名称;version_code: 配置文件版本号;version_info: 配置文件版本信息;article-alias: 文章别名(OPTIONAL), 生成的目录格式为[bbs|json]_YYmmdd[_alias];sn-url-prefix: 文章的地址格式, 一般格式为http://xxx.yyy.zzz/post-xxx-yyy-{}.shtml;page-start: 页面开始;page-stop: 页面结束(OPTIONAL);page-flag: 页面标记(OPTIONAL), 支持4 种类型,REGION_INCLUSIVE_EXCLUSIVE,REGION_EXCLUSIVE_INCLUSIVE,REGION_EXCLUSIVE_EXCLUSIVE,REGION_INCLUSIVE_INCLUSIVE;article-flag: 文章数据流生成风格(OPTIONAL);article-multi-media-persistence: 文章关联到的媒体资源持久化到本地(OPTIONAL,RECOMMENDED);article-multi-media-quality: 文章关联到的媒体资源质量(OPTIONAL);User-Agent: 文章请求的用户代理(OPTIONAL，RECOMMENDED);output-directory: 文章输出到指定目录(OPTIONAL，RECOMMENDED);output-uuid-encryptor: 文章输出 id 加密处理(OPTIONAL);1.3. CLI 配置$dvs-sniffer-h
usage:dvs-sniffer[-h][-V][-amp][-amq][-a2[article-alias]][-ad[article-describe]][-af][-rs[region-start]][-re[region-end]][-rm[region-mask]][-due][-ua[User-Agent]]sn-url[destination-directory]thisisadvsnetworksnifferexecutionprogram.thesnifferdestinationurlformatmustconformtothefollowingcontinuousURLs:eg:1.http://bbs.xxx.cn/list-xyz-1.shtml2.http://bbs.xxx.cn/list-xyz-2.shtml3.http://bbs.xxx.cn/list-xyz-3.shtml4....5.http://bbs.xxx.cn/list-xyz-{}.shtml


positionalarguments:sn-urlthesnifferdestinationurl.destination-directorythesnifferdestinationdirectory.

optionalarguments:-h,--helpshowthishelpmessageandexit-V,--versiontheshowversionandexit.-amp,--article-multi-media-persistenceifTrue:multimediaresourcesarestoredlocally,otherwisetheyarenot,andthedefaultvalueisTrue.-amq,--article-multi-media-qualityifTrue:multimediaresourcesarehighquality,otherwisetheyarenot,andthedefaultvalueisFalse.-a2[article-alias],--article-alias[article-alias]ashorttextarticlealiasofthesniffertobe.-ad[article-describe],--article-describe[article-describe]ashorttextarticledescriptionofthesniffertobe.-af,--article-flagifFalse:firstpull,secondtranslateTrue:onepullafteranothertranslate,andthedefaultvalueisTrue.-rs[region-start],--region-start[region-start]abrieflydescribetherangestarttobesniffedmathematically.-re[region-end],--region-end[region-end]abrieflydescribetherangeendtobesniffedmathematically.-rm[region-mask],--region-mask[region-mask]Theolfactoryspatialrangeofthesniffercanonlybethefollowingvalues:REGION_INCLUSIVE_EXCLUSIVE=0,REGION_EXCLUSIVE_INCLUSIVE=1,REGION_EXCLUSIVE_EXCLUSIVE=2,REGION_INCLUSIVE_INCLUSIVE=3,andthedefaultvalueisREGION_INCLUSIVE_EXCLUSIVE.-due,--destination-uuid-encryptorthesnifferdestinationuuidencryptor,andthedefaultvalueisTrue.-ua[User-Agent],--user-agent[User-Agent]theuseragentflagofsetsnifferfordefaultnetworkaccess,whichisthemacintoshsystemidentifierbydefault.

thecopyrightbelongstodvsthatreservetherightoffinalinterpretation.二. 运行2.1. 脚本运行脚本运行如下:# Windowspython./scripts/crawler_script.py# Macintoshpython.\scripts\crawler_script.py2.1. CLI 运行# Windows and Macintosh# region: [1, 2)dvs-sniffer-amq-rs1-re2http://xxx.yyy.zzz/post-1.html# the default destination directorydvs-sniffer-amq-rs1-re2http://xxx.yyy.zzz/post-1.html\var\...\dvs-sniffer\# the special destination directory