Package: iparse
iparseiparseis a Python package for parsing HTML to structured data in an easy way with as little code as possible.It aims to make the process of parsing HTML quick and easy!iparse highlights:mainly code withYAMLonly refine raw HTML info withfewest python codelot's HTML layout changes, onlyYAMLwill be involvedInstallationpipinstalliparseA Simple Examplefor HTML page: i.e.lovely xkcd pythonto get the structured data all you need arecreate a class inherit fromIParserwrite a YAML config file represents all locatorscreatexkcd_353.pyxkcd_353.pywill go through the startup_dir, look for a file named as the snake_case of the ClassName withoutsuffix:Parser, soXkcdParserwill bexkcd.yamlfrompathlibimportPathfromiparse._parseimportIParser,RsvWordsHOME_DIR=Path(__file__).parents[0]classXkcdParser(IParser):def__init__(self,file_name,is_test_mode=False,**kwargs):kwargs['startup_dir']=kwargs.get('startup_dir',HOME_DIR)super().__init__(file_name,is_test_mode=is_test_mode,**kwargs)if__name__=="__main__":xkcd=XkcdParser(file_name=HOME_DIR/'xkcd_python_353.htm')xkcd.do_parse()print(xkcd.data)create a file namedxkcd.yamlyou can use any locator that is supported, butcss selectoris recommendedpage:# css_selector of title: head>titletitle:head>title# css_selector: div#footnotefootnote:div#footnote# css_selector: div#licenseTextlicense:div#licenseTextthe output parsed datathe parsed dataxkcd.datais dict, but you can also use it withxkcd.data_as_yaml/xkcd.data_as_jsonyaml outputpage:footnote:"xkcd.comisbestviewedwithNetscapeNavigator4.0orbelowonaPentium\\ 3\xB11emulatedinJavascriptonanAppleIIGSatascreenresolutionof1024x1.\\ Pleaseenableyouradblockers,disablehigh-heatdrying,andremoveyourdevicefrom\\ AirplaneModeandsetittoBoatMode.Forsecurityreasons,pleaseleavecaps\\ lockonwhilebrowsing."license:'ThisworkislicensedunderaCreativeCommonsAttribution-NonCommercial2.5License.Thismeansyou''refreetocopyandsharethesecomics(butnottosellthem).Moredetails.'title:'xkcd:Python'json output{"page":{"footnote":"xkcd.com is best viewed with Netscape Navigator 4.0 or below on a Pentium 3\u00b11 emulated in Javascript on an Apple IIGSat a screen resolution of 1024x1. Please enable your ad blockers, disable high-heat drying, and remove your devicefrom Airplane Mode and set it to Boat Mode. For security reasons, please leave caps lock on while browsing.","license":"\n\nThis work is licensed under a\nCreative Commons Attribution-NonCommercial 2.5 License.\n\nThis means you're free to copy and share these comics (but not to sell them). More details.\n","title":"xkcd: Python"}}Details# all settings added to __raw, will be kept as it added__raw:site_url:https://xkcd.com/page:# if not _locator supplied will reuse parent soup# page has no parent soup, so use default root souptitle:head>titlefootnote:div#footnotelicense:_locator:div#licenseText# strip blank with true, but also can specified a str_striped:truetop_container:# we set a _locator here, all sub-nodes will select within top_container_locator:div#topContainertop_left:# _index:~ means None, so we can use whole list_index:~_locator:div#topLeft>ul>li>a# if non-reserved key set to ~, means use parent soup, and use its text# this is a convenient way to get textmenu_text:~menu_url:# when other attributes exist, no need to add _locator to use its parent soup_attr:href# if we need some extra work on _attr, goes with two ways# 1. `_attr_refine: true` will auto generate => _refine_menu_url_href# the rule of auto-generator is _refine_<key_name>_<attr_value># 2. `_attr_refine: _a_valid_method_name`_attr_refine:truetop_right:_locator:div#topRightmasthead:# two way to get more than one attributes on a element# e.g. image.src/.alt# way1: if all src/alt need refine, this will treat attrs as listimage_1:_attr:-src-alt_attr_refine:true_locator:&LOGO_IMGspan>a>img# way2: not all src/alt need refine, this will treat attrs as dictimage_2:_locator:*LOGO_IMGsrc:_attr:src# only set _attr_refine to src# 1. _attr_refine: true => _refine_src_src# 2. _attr_refine: _refine_image_1_src to reuse exists method_attr_refine:_refine_image_1_srcalt:_attr:altslogan:span#sloganmoreplease check thetests/for more infomation.