Package: tp-html
Template Parser HTMLThis tool can help get useful data from html web page. It parses html page with the template file which marks data that you need with special attributes. The template file positions html blocks that contain data, and describes types, names and structures of data. You can modify an example page file to get it, or write a basic html structure that can position your data. I suggest you to use the first method, the tool can delete irrelevant parts and organize html tree automatically.installpip install tp_htmlHow to usefromtp_htmlimportTemplate,ThtmlParser# get templatetemplate=Template(template_file='samples/basic_template.html')# save templatetemplate.save('samples/basic_template.min.html')# get parserparser=ThtmlParser(template_file='samples/basic_template.html')parser=ThtmlParser(template_text='...')parser=ThtmlParser(template=template)# parse datadata=parser.parse(page_file='samples/basic_sample.html',encoding='urf-8')data=parser.parse(page_url='http://.....')data=parser.parse(page_text='.....')Template filestringTo get data from content or attributes of element.<ahref="....">link</a>To get content. This will get data {'name': 'link'}<ap-value="true"p-name="name"></a>To get href. This will get data {'name': '...'}<ap-value="true"p-name="name"p-item="href"></a>listFor HTML<ulclass="image-list"><liclass="image-item"><ahref="/image/1"></a></li><liclass="image-item"><ahref="/image/2"></a></li><liclass="image-item"><ahref="/image/3"></a></li><liclass="image-item"><ahref="/image/4"></a></li></ul>template:<ulclass="image-list"p-value="true"p-name="images"p-type="list"><liclass="image-item"><ap-value="true"p-item="href"></a></li></ul>data:{"images":["/image/1","/image/2","/image/3","/image/4"]}In list template, in the element which is marked with p-type=list, require one child p-value node and just one that is for selecting item data. If list item is dict or list, structure in item is also allowed.dictFor HTML<divclass="dict-data-container"><aclass="user-name"href="/user/13456"title="user xxx">xxx</a><pclass="user-age">20</p><divclass="sub-div"><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div></div>template:<divclass="dict-data-container"p-value="true"p-name="user_link"p-type="dict"><aclass="user-name"p-value="true"p-name="name link title"p-item="string href title"></a><pclass="user-age"p-value="true"p-name="age"></p><divclass="sub-div"><pclass="user-fans-num"p-value="true"p-name="fans_num"></p><pclass="user-follow-num"p-value="true"p-name="follow_num"></p></div></div>data:{"user_link":{"name":"xxx","link":"/user/13456","title":"user xxx","age":"20","fans_num":"10","follow_num":"20"}}In dict template, p-name is required for key of dictionary. Multiple p-item is allowed, split with space, and "string" means content of element, others items are attributies name.complex nestinghtml<!DOCTYPE html><htmllang="en"><head><metacharset="UTF-8"><title>Title</title></head><body><divid="container"><divclass="column-1"><!--dict in list--><ulclass="column-1-ul"><li><divclass="dict-data-container"><aclass="user-name"href="/user/1"title="user xxx">xxx</a><pclass="user-age">20</p><divclass="sub-div"><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div></div></li><li><divclass="dict-data-container"><aclass="user-name"href="/user/2"title="user yyy">yyy</a><pclass="user-age">10</p><divclass="sub-div"><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div></div></li></ul></div><divclass="column-2"><!--list in dict--><divclass="dict-data-container"><aclass="user-name"href="/user/1"title="user xxx">xxx</a><pclass="user-age">20</p><divclass="sub-div"><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div><ulclass="image-list"><liclass="image-item"><ahref="/image/1"></a></li><liclass="image-item"><ahref="/image/2"></a></li><liclass="image-item"><ahref="/image/3"></a></li><liclass="image-item"><ahref="/image/4"></a></li></ul></div></div><divclass="column-3"><!--dict in dict--><divclass="dict-data-container"><divclass="profile"><aclass="user-name"href="/user/1"title="user xxx">xxx</a><pclass="user-age">20</p></div><divclass="sub-div"><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div></div></div><divclass="column-4"><!--list in list--><ulclass="image-list"><ulclass="image-tags"><pclass="tag">tag1-1</p><pclass="tag">tag1-2</p><pclass="tag">tag1-3</p></ul><ulclass="image-tags"><pclass="tag">tag2-1</p><pclass="tag">tag2-2</p><pclass="tag">tag2-3</p></ul><ulclass="image-tags"><pclass="tag">tag3-1</p><pclass="tag">tag3-2</p><pclass="tag">tag3-3</p></ul></ul></div></div></body></html>template:<divid="container"><divclass="column-1"><!--dict in list--><ulclass="column-1-ul"p-value="true"p-name="user_list"p-type="list"><lip-value="true"p-type="dict"><divclass="dict-data-container"><aclass="user-name"p-value="true"p-name="name link title"p-item="string href title"></a><pclass="user-age"p-value="true"p-name="age"></p><divclass="sub-div"><pclass="user-fans-num"p-value="true"p-name="fans_num"></p><pclass="user-follow-num"p-value="true"p-name="follow_num"></p></div></div></li></ul></div><divclass="column-2"><!--list in dict--><divclass="dict-data-container"p-value="true"p-name="user_all"p-type="dict"><aclass="user-name"p-value="true"p-name="name link title"p-item="string href title"></a><pclass="user-age"p-value="true"p-name="age"></p><divclass="sub-div"><pclass="user-fans-num"p-value="true"p-name="fans_num"></p><pclass="user-follow-num"p-value="true"p-name="follow_num"></p></div><ulclass="image-list"p-value="true"p-name="images"p-type="list"><liclass="image-item"><ap-value="true"p-item="href"></a></li></ul></div></div><divclass="column-3"><!--dict in dict--><divclass="dict-data-container"p-value="true"p-name="user_info"p-type="dict"><divclass="profile"p-value="true"p-name="profile"p-type="dict"><aclass="user-name"p-value="true"p-name="name link title"p-item="string href title"></a><pclass="user-age"p-value="true"p-name="age"></p></div><divclass="sub-div"p-value="true"p-name="counts"p-type="dict"><pclass="user-fans-num"p-value="true"p-name="fans_num"></p><pclass="user-follow-num"p-value="true"p-name="follow_num"></p></div></div></div><divclass="column-4"><!--list in list--><ulclass="image-list"p-value="true"p-name="image_tags"p-type="list"><ulclass="image-tags"p-value="true"p-type="list"><pclass="tag"p-value="true"></p></ul></ul></div></div>data:{"user_list":[{"name":"xxx","link":"/user/1","title":"user xxx","age":"20","fans_num":"10","follow_num":"20"},{"name":"yyy","link":"/user/2","title":"user yyy","age":"10","fans_num":"10","follow_num":"20"}],"user_all":{"name":"xxx","link":"/user/1","title":"user xxx","age":"20","fans_num":"10","follow_num":"20","images":["/image/1","/image/2","/image/3","/image/4"]},"user_info":{"profile":{"name":"xxx","link":"/user/1","title":"user xxx","age":"20"},"counts":{"fans_num":"10","follow_num":"20"}},"image_tags":[["tag1-1","tag1-2","tag1-3"],["tag2-1","tag2-2","tag2-3"],["tag3-1","tag3-2","tag3-3"]]}p-data tag in templatep-data is a empty html tag using in template to organize data structure. In the example below, four elements are in the same level. But with p-data tag, they can have more structure.HTML:<divclass="dict-data-container"><aclass="user-name"href="/user/1"title="user xxx">xxx</a><pclass="user-age">20</p><pclass="user-fans-num">10</p><pclass="user-follow-num">20</p></div>template:<divclass="dict-data-container"p-value="true"p-name="user_info"p-type="dict"><p-datap-value="true"p-name="profile"p-type="dict"><aclass="user-name"p-value="true"p-name="name link title"p-item="string href title"></a><pclass="user-age"p-value="true"p-name="age"></p></p-data><p-datap-value="true"p-name="counts"p-type="dict"><pclass="user-fans-num"p-value="true"p-name="fans_num"></p><pclass="user-follow-num"p-value="true"p-name="follow_num"></p></p-data></div>data:{"user_info":{"profile":{"name":"xxx","link":"/user/1","title":"user xxx","age":"20"},"counts":{"fans_num":"10","follow_num":"20"}}}Save templateThe tool also provide a method to save minimal template into a file. It will have a faster template building speed with the minimal template file.template=TemplateParser(template_file='samples/pixiv_user_template.html')template.save('samples/pixiv_user_template.min.html')parser=WebPageParser(template_file='samples/pixiv_user_template.min.html')data=parser.parse(page_file='samples/pixiv_user.html')minimal template:<p-dataselector="html > body > div#wrapper > div.layout-a"><p-dataselector="div.layout-column-2 > div._unit > div.works_area.profile > div.works_info > div.worksOption.profile-page > div.worksListOthers > div.works-illust > ul._image-items.no-user"p-value="true"p-type="list"p-name="image-list"><p-dataselector="li.image-item"p-value="true"p-type="dict"><p-dataselector="a.work._work"p-value="true"p-name="url"p-item="href"><p-dataselector="div._layout-thumbnail > img._thumbnail.ui-scroll-view"p-value="true"p-name="illust_id tags"p-item="data-id data-tags"></p-data></p-data></p-data></p-data><p-dataselector="div.layout-column-1 > div.ui-layout-west > div._user-profile-card > div.profile > a.user-name"p-value="true"p-type="string"p-name="name"></p-data></p-data>TestTime test for templates in samples folder. Test tool istimefunctest code:parser=ThtmlParser(template_file='samples/basic_template.html')functime.func_time(ThtmlParser,template_file='samples/basic_template.html')functime.func_time(parser.parse,page_file='samples/basic_sample.html')result:basic_template.html
TemplateParser AVG(1000): 1.251ms
parse AVG(100): 3.2599ms

complex_template.html
TemplateParser AVG(1000): 1.6005ms
parse AVG(100): 6.265ms

pixiv_user_template.html
TemplateParser AVG(10): 24.7ms
parse AVG(10): 34.4997ms

pixiv_user_template.min.html
TemplateParser AVG(1000): 760.001183us
parse AVG(10): 31.2003ms