Package: py-parse
READMEA simplest html parsing library.Key features:no third-party dependenciesno need to know CSS, Xpath or complicated rules to find elementinteraction with native python lambda syntax or function-predicateopportunity to work with damaged htmlability to use element relations (find ancestor, descendant, siblings)a standard find first element or find all by current filterInstallationVia pip:pip install py_parseFirst exampleLets get src attribute (link) of the Google logo on google.comimportrequestsfrompy_parseimportParser# get content of the google web pagecontent=requests.get('https://www.google.com/').text# find first element with img-tag and 'alt' attribute equal to Google (logo)google_logo=Parser().parse(content).find(lambdae:e.tag=='img'ande.alt=='Google')# prints src attribute of the logo elementprint(google_logo.src)You will see following result/images/branding/googlelogo/1x/googlelogo_white_background_color_272x92dp.pngIf there is no element with current filter, you will get exception with filter text (if lambda was used):
For code above lets say we use wrong filtergoogle_logo=Parser().parse(content).find(lambdae:e.tag=='img'ande.alt=='Wrong')You will see following result...traceback...
py_parse.exceptions.NoSuchElementError: No elements with current filter (e.tag == 'img' and e.alt == 'Wrong')HOW IT WORKSDuring parsing all html-elements in DOM converts to Node objects, which remains all relations
(parent, child, sibling) and get all their attributes from html-element.For example<divclass="some"type="submit">My text</div>will return (after parsing):frompy_parseimportParserelement='<div class="some" type="submit">My text</div>'html_element=Parser().parse(element)[0]print(html_element.text)# My textprint(html_element.tag)# divprint(html_element.class_)# someAs you can see, all html attributes became object attributes, so you can use it in your filtersBut remember:Attribute tag is required, always present and can't be NoneAttribute text always present BUT can be NoneAttribute class became class_ in an object (html_element.class_), and it is not requiredAs you know, web page is a hierarchy, where html is a ancestor for all elements, and they all are nested in html.
Function parse returns Nodes object, that is just container (like list) for Node objects. And for most cases that Nodes
will have just one element (html), which contains all other elements inside (nested). So, for using search, you need to
use methods like find or find_all of the Nodes.The main idea is simple parsing without knowledge of CSS or XPATH, user need to know just a basic part of html. If you
can understand what is html-element, tag, attribute, parent, child -you can parse all that you need.Text - is that we looking forUser parse html to get some text of other values from it. Attributes is simple as get something from any python object.
For example, when we parse following html<divclass="some"value="text"></div>we can get value simple by using attribute:value=Parser().parse('<div class="some" value="text"></div>').find_tag('div').valueprint(value)# textIf text of the element is split by other tags, it will be split by newline character in resulting text. For example div
text had some br's inside:<div>One<br>Two<br>Three</div>result will betext_with_n=Parser().parse('<div>One<br>Two<br>Three</div>').find_tag('div').textprint(text_with_n)# One\nTwo\nThreetexts=text_with_n.split('\n')print(texts)# ['One', 'Two', 'Three']And finally we can get cleared text - the text without newlines characters, tabs or leading/following spaces.
so following html<div>One<br>Two<br>Three</div>transforms toclear_text=Parser().parse('<div>  One<br>\nTwo<br>\tThree   </div>').find_tag('div').cleared_text()print(clear_text)# OneTwoThreeFind methodsMethodfind_allof the Nodes objects returns all found elements. If you not specify a filter, then all elements be
in result. With filter, you get only elements, that satisfying the condition in it. If there are no such elements, then
empty Nodes container returns.Method find based onfind_all, but returns just first element with that filter. If there are no results, then
exception will be raised.Methodsfind_tagsandfind_tagare similar tofind_all,find, but it makes search with tag more
convinient. For examplefind_all(lambdae:e.tag=='div')is equivalent tofind_tags('div')Simple FilteringFor all examples we will use content of the python documentation pagehttps://docs.python.org/3/So, start of all code isimportrequestsfrompy_parseimportParsercontent=requests.get('https://docs.python.org/3/').text1. Find by tagLet's find first element with 'strong' tag and get it textfirst_strong=Parser().parse(content).find(lambdae:e.tag=='strong')print(first_strong.text)# Parts of the documentation:or you can use specialfind_tagmethod:first_strong=Parser().parse(content).find_tag('strong')print(first_strong.text)# Parts of the documentation:2. Find by tag and text (always present in any element)tables=Parser().parse(content).find(lambdae:e.tag=='strong'ande.textande.text=='Indices and tables:')print(tables.text)# Indices and tables:ortables=Parser().parse(content).find_tag('strong',lambdae:e.text=='Indices and tables:')print(tables.text)# Indices and tables:3. Find by containing textcopyright_=Parser().parse(content).find(lambdae:'pyri'ine.text)# pyri is a part of Copyrightprint(copyright_)# <a class="biglink" href="copyright.html">Copyright</a>In this example we print Node object itself, but not its text attribute.4. Find element which has idelement_with_id=Parser().parse(content).find(lambdae:'id'ine)# 'id' in e - checks element has "id" attributeprint(element_with_id)# <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>5. Find element by tag and type, then get it valueLet's find 'Go' button to search on documentation page# finds element with input tag, which has type and this type equal to submitgo=Parser().parse(content).find(lambdae:e.tag=='input'ande.type=='submit')print(go.value)# Goorgo=Parser().parse(content).find_tag('input',lambdae:e.type=='submit')print(go.value)# Go6. Finds all script elementsscripts=Parser().parse(content).find_all(lambdae:e.tag=='script')# Using find_all to finds all elementsforscriptinscripts:print(script)# <script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script># <script src="_static/jquery.js"></script># <script src="_static/underscore.js"></script># <script src="_static/doctools.js"></script># <script src="_static/language_data.js"></script># <script src="_static/sidebar.js"></script># <script type="text/javascript" src="_static/copybutton.js"></script># <script type="text/javascript">$('.inline-search').show(0);</script># <script type="text/javascript">$('.inline-search').show(0);</script># <script type="text/javascript" src="_static/switchers.js"></script>or with specialfind_tagsmethod:scripts=Parser().parse(content).find_tags('script')# Using find_tags to finds all elements with that tagforscriptinscripts:print(script)7. Finds all elements by part of the class namesphinxes=Parser().parse(content).find_all(lambdae:'sphinx'ine.class_)# finds all elements containing 'sphinx' in class nameforsphinxinsphinxes:print(sphinx)# <div class="sphinxsidebar" role="navigation" aria-label="main navigation"></div># <div class="sphinxsidebarwrapper"></div>8. Using function-predicateSometimes your search filter became too long and heavy to use it in lambda, in that cases you should create a function-predicate
with clear name to use in filters!defimages_with_alt_and_style(e)->bool:# predicate function (takes html element and returns bool)returne.tag=='img'and'alt'ineand'middle'ine.styleli_s=Parser().parse(content).find_all(images_with_alt_and_style)forliinli_s:print(li)# <img src="_static/py.png" alt="" style="vertical-align: middle; margin-top: -1px"></img># <img src="_static/py.png" alt="" style="vertical-align: middle; margin-top: -1px"></img>Relations filteringContact meLexman2@yandex.ru