Package: uniparser
uniparserProvides a general low-code page parsing solution.Backwards Compatibility Breaking Warning:uniparserwill not install any default parsers after version v3.0.0. You can install some of them manually ('selectolax', 'jsonpath-rw-ext', 'objectpath', 'bs4', 'toml', 'pyyaml>=5.3', 'lxml', 'jmespath'). This warning will keep 2 versions.Installpip install uniparser -Uorpip install uniparser[parsers]with default 3rd parsersWhy?Reduced the code quantity from plenty of similar crawlers & parsers.  Don't Repeat Yourself.Make the parsing process of different parsers persistent.Separating the crawler code from main app code, no need to redeploy app when adding a new crawler.Provide a universal solution for crawler platforms.Summarize common string parsing tools on the market.The implementation of web views is to be plug-in and portable, which means it can be mounted on other web apps as asub_app:app.mount("/uniparser", uniparser_app)Here is thelow-codeweb UI screenshot.Feature ListSupport most of popular parsers for HTML / XML / JSON / AnyString / Python objectParser docsparser list1. css (HTML)
      1. bs4
  2. xml
      1. lxml
  3. regex
  4. jsonpath
      1. jsonpath-rw-ext
  5. objectpath
      1. objectpath
  6. jmespath
      1. jmespath
  7. time
  8. loader
      1. json / yaml / toml
          1. toml
          2. pyyaml
  9. udf
      1. source code for exec & eval which named as **parse**
  10. python
      1. some  common python methods, getitem, split, join...
  11. *waiting for new ones...*Request args persistence, support curl-string, single-url, dict, json.A simple Web UI for generate & test CrawlerRule.Serializable JSON rule class for saving the whole parsing process.Each ParseRule / CrawlerRule / HostRule subclass can be json.dumps to JSON for persistence.Therefore, they also can be loaded from JSON string.Nest relation of rule names will be treat as the result format. (Rule's result will be ignore if has childs.)Rule ClassesJsonSerializableis the base class for all the rules.dumps classmethod can dump self as a standard JSON string.loads classmethod can load self from a standard JSON string, which means the new object will has the methods as a rule.ParseRuleis the lowest level for a parse mission, which contains how to parse a input_object. Sometimes it also has a list of ParseRule as child rules.Parse result is a dict that rule_name as key and result as value.CrawlerRulecontains some ParseRules, which has 3 attributes besides the rule name:request_args tell the http-downloader how to send the request.parse_rules is a list of ParseRule, and the parsing result format is like {CrawlerRule_name: {ParseRule1['name']: ParseRule1_result, ParseRule2['name']: ParseRule2_result}}.regex tells how to find the crawler_rule with a given url.HostRulecontains a dict like: {CrawlerRule['name']: CrawlerRule}, with thefindmethod it can get the specified CrawlerRule with a given url.JSONRuleStorageis a simple storage way, which saved the HostRules in a JSON file. On the production environment this is not a good choice, maybe redis / mysql / mongodb can give a hand.Uniparseris the center console for the entire crawler process. It handled download middleware, parse middleware. Detail usage can be find atuniparser.crawler.Crawler, or have a loot at [Quick Start].For custom settings, such as json loader, please update the uniparser.config.GlobalConfig.Quick StartMission: Crawl pythonMeta-PEPsOnly less than 25 lines necessary code besides the rules(which can be saved outside and auto loaded).HostRules will be saved at$HOME/host_rules.jsonby default, not need to init every time.CrawlerRule JSON & Expected Result# These rules will be saved at `$HOME/host_rules.json`crawler=Crawler(storage=JSONRuleStorage.loads(r'{"www.python.org": {"host": "www.python.org", "crawler_rules": {"main": {"name":"list","request_args":{"method":"get","url":"https://www.python.org/dev/peps/","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"}},"parse_rules":[{"name":"__request__","chain_rules":[["css","#index-by-category #meta-peps-peps-about-peps-or-processes td.num>a","@href"],["re","^/","@https://www.python.org/"],["python","getitem","[:3]"]],"childs":""}],"regex":"^https://www.python.org/dev/peps/$","encoding":""}, "subs": {"name":"detail","request_args":{"method":"get","url":"https://www.python.org/dev/peps/pep-0001/","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"}},"parse_rules":[{"name":"title","chain_rules":[["css","h1.page-title","$text"],["python","getitem","[0]"]],"childs":""}],"regex":"^https://www.python.org/dev/peps/pep-\\d+$","encoding":""}}}}'))expected_result={'list':{'__request__':['https://www.python.org/dev/peps/pep-0001','https://www.python.org/dev/peps/pep-0004','https://www.python.org/dev/peps/pep-0005'],'__result__':[{'detail':{'title':'PEP 1 -- PEP Purpose and Guidelines'}},{'detail':{'title':'PEP 4 -- Deprecation of Standard Modules'}},{'detail':{'title':'PEP 5 -- Guidelines for Language Evolution'}}]}}The Whole Source CodefromuniparserimportCrawler,JSONRuleStorageimportasynciocrawler=Crawler(storage=JSONRuleStorage.loads(r'{"www.python.org": {"host": "www.python.org", "crawler_rules": {"main": {"name":"list","request_args":{"method":"get","url":"https://www.python.org/dev/peps/","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"}},"parse_rules":[{"name":"__request__","chain_rules":[["css","#index-by-category #meta-peps-peps-about-peps-or-processes td.num>a","@href"],["re","^/","@https://www.python.org/"],["python","getitem","[:3]"]],"childs":""}],"regex":"^https://www.python.org/dev/peps/$","encoding":""}, "subs": {"name":"detail","request_args":{"method":"get","url":"https://www.python.org/dev/peps/pep-0001/","headers":{"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36"}},"parse_rules":[{"name":"title","chain_rules":[["css","h1.page-title","$text"],["python","getitem","[0]"]],"childs":""}],"regex":"^https://www.python.org/dev/peps/pep-\\d+$","encoding":""}}}}'))expected_result={'list':{'__request__':['https://www.python.org/dev/peps/pep-0001','https://www.python.org/dev/peps/pep-0004','https://www.python.org/dev/peps/pep-0005'],'__result__':[{'detail':{'title':'PEP 1 -- PEP Purpose and Guidelines'}},{'detail':{'title':'PEP 4 -- Deprecation of Standard Modules'}},{'detail':{'title':'PEP 5 -- Guidelines for Language Evolution'}}]}}deftest_sync_crawler():result=crawler.crawl('https://www.python.org/dev/peps/')print('sync result:',result)assertresult==expected_resultdeftest_async_crawler():asyncdef_test():result=awaitcrawler.acrawl('https://www.python.org/dev/peps/')print('sync result:',result)assertresult==expected_resultasyncio.run(_test())test_sync_crawler()test_async_crawler()Uniparser Rule Test Console (Web UI)pip install bottle uniparserpython -m uniparser 8080open browser =>http://127.0.0.1:8080/Download URLParse HTMLShow result as JSON{"CSS demo":{"Tag attribute":"post"}}As we can see, CrawlerRule's name is the root key, and ParseRule's name as the others.Async environment usage:Fastapiimportuvicornfromuniparser.fastapi_uiimportappif__name__=="__main__":uvicorn.run(app,port=8080)# http://127.0.0.1:8080or Fastapi subapp usageimportuvicornfromfastapiimportFastAPIfromuniparser.fastapi_uiimportappassub_appapp=FastAPI()app.mount('/uniparser',sub_app)if__name__=="__main__":uvicorn.run(app,port=8080)# http://127.0.0.1:8080/uniparser/More UsageSome Demos:Click the dropdown buttons on top of the Web UITest Code:test_parsers.pyAdvanced Usage:Create crawler ruleforwatchdogsGenerate parsers docfromuniparserimportUniparserforiinUniparser().parsers:print(f'##{i.__class__.__name__}({i.name})\n\n```\n{i.doc}\n```')BenchmarkCompare parsers and choose a faster onecss:2558calls/sec,['<a class="url" href="/">title</a>','a.url','@href']css:2491calls/sec,['<a class="url" href="/">title</a>','a.url','$text']css:2385calls/sec,['<a class="url" href="/">title</a>','a.url','$innerHTML']css:2495calls/sec,['<a class="url" href="/">title</a>','a.url','$html']css:2296calls/sec,['<a class="url" href="/">title</a>','a.url','$outerHTML']css:2182calls/sec,['<a class="url" href="/">title</a>','a.url','$string']css:2130calls/sec,['<a class="url" href="/">title</a>','a.url','$self']=================================================================================css1:2525calls/sec,['<a class="url" href="/">title</a>','a.url','@href']css1:2402calls/sec,['<a class="url" href="/">title</a>','a.url','$text']css1:2321calls/sec,['<a class="url" href="/">title</a>','a.url','$innerHTML']css1:2256calls/sec,['<a class="url" href="/">title</a>','a.url','$html']css1:2122calls/sec,['<a class="url" href="/">title</a>','a.url','$outerHTML']css1:2142calls/sec,['<a class="url" href="/">title</a>','a.url','$string']css1:2483calls/sec,['<a class="url" href="/">title</a>','a.url','$self']=================================================================================selectolax:15187calls/sec,['<a class="url" href="/">title</a>','a.url','@href']selectolax:19164calls/sec,['<a class="url" href="/">title</a>','a.url','$text']selectolax:19699calls/sec,['<a class="url" href="/">title</a>','a.url','$html']selectolax:20659calls/sec,['<a class="url" href="/">title</a>','a.url','$outerHTML']selectolax:20369calls/sec,['<a class="url" href="/">title</a>','a.url','$self']=================================================================================selectolax1:17572calls/sec,['<a class="url" href="/">title</a>','a.url','@href']selectolax1:19096calls/sec,['<a class="url" href="/">title</a>','a.url','$text']selectolax1:17997calls/sec,['<a class="url" href="/">title</a>','a.url','$html']selectolax1:18100calls/sec,['<a class="url" href="/">title</a>','a.url','$outerHTML']selectolax1:19137calls/sec,['<a class="url" href="/">title</a>','a.url','$self']=================================================================================xml:3171calls/sec,['<dc:creator><![CDATA[author]]></dc:creator>','creator','$text']=================================================================================re:220240calls/sec,['a a b b c c','a|c','@b']re:334206calls/sec,['a a b b c c','a','']re:199572calls/sec,['a a b b c c','a (a b)','$0']re:203122calls/sec,['a a b b c c','a (a b)','$1']re:256544calls/sec,['a a b b c c','b','-']=================================================================================jsonpath:28calls/sec,[{'a':{'b':{'c':1}}},'$..c','']=================================================================================objectpath:42331calls/sec,[{'a':{'b':{'c':1}}},'$..c','']=================================================================================jmespath:95449calls/sec,[{'a':{'b':{'c':1}}},'a.b.c','']=================================================================================udf:58236calls/sec,['a b c d','input_object[::-1]','']udf:64846calls/sec,['a b c d','context["key"]',{'key':'value'}]udf:55169calls/sec,['a b c d','md5(input_object)','']udf:45388calls/sec,['["string"]','json_loads(input_object)','']udf:50741calls/sec,['["string"]','json_loads(obj)','']udf:48974calls/sec,[['string'],'json_dumps(input_object)','']udf:41670calls/sec,['a b c d','parse = lambda input_object: input_object','']udf:31930calls/sec,['a b c d','def parse(input_object): context["key"]="new";return context',{'key':'new'}]=================================================================================python:383293calls/sec,[[1,2,3],'getitem','[-1]']python:350290calls/sec,[[1,2,3],'getitem','[:2]']python:325668calls/sec,['abc','getitem','[::-1]']python:634737calls/sec,[{'a':'1'},'getitem','a']python:654257calls/sec,[{'a':'1'},'get','a']python:642111calls/sec,['a b\tc\n\td','split','']python:674048calls/sec,[['a','b','c','d'],'join','']python:478239calls/sec,[['aaa',['b'],['c','d']],'chain','']python:191430calls/sec,['python','template','1 $input_object 2']python:556022calls/sec,[[1],'index','0']python:474540calls/sec,['python','index','-1']python:619489calls/sec,[{'a':'1'},'index','a']python:457317calls/sec,['adcb','sort','']python:494608calls/sec,[[1,3,2,4],'sort','desc']python:581480calls/sec,['aabbcc','strip','a']python:419745calls/sec,['aabbcc','strip','ac']python:615518calls/sec,['\ta ','strip','']python:632536calls/sec,['a','default','b']python:655448calls/sec,['','default','b']python:654189calls/sec,[' ','default','b']python:373153calls/sec,['a','base64_encode','']python:339589calls/sec,['YQ==','base64_decode','']python:495246calls/sec,['a','0','b']python:358796calls/sec,['','0','b']python:356988calls/sec,[None,'0','b']python:532092calls/sec,[{0:'a'},'0','a']=================================================================================loader:159737calls/sec,['{"a": "b"}','json','']loader:38540calls/sec,['a = "a"','toml','']loader:3972calls/sec,['animal: pets','yaml','']loader:461297calls/sec,['a','b64encode','']loader:412507calls/sec,['YQ==','b64decode','']=================================================================================time:39241calls/sec,['2020-02-03 20:29:45','encode','']time:83251calls/sec,['1580732985.1873155','decode','']time:48469calls/sec,['2020-02-03T20:29:45','encode','%Y-%m-%dT%H:%M:%S']time:74481calls/sec,['1580732985.1873155','decode','%b%d%Y %H:%M:%S']TasksRelease to pypi.orgUpload dist with Web UIAddgithub actionsfor testing packageWeb UI for testing rulesComplete the doc in detailCompare each parser's performance