Package: webparsa
webparsaThis project uses XML templates to extract data from websites for you, with almost no code.XML templates are used to mimic the structure of the HTML itself, allowing you to make intuitive selectors.
You could literally copy and paste website code, and specify which attributes are the variables you want, and it would
work.Storing single valuesNote: for images, use the<p_img>tag, because img tags can't have children in HTML.To extract a certain value from a part of the element, use the<value>tag.
Value tags need two things:name: the name of the variable to store under(inner text): the attribute to store.Storing lists of valuesTo import a list of similar divs as a python list, wrap the single div that encloses the tags with<list>tags.
List tags need only one attribute:name: the name of the variable to store under
Doesn't have to be the direct parent of a value!Storing dicts of valuesTo group some values together as a dict, wrap the values in a<dict>tag.
Requires a name attribute, like<list>and<value>.
Doesn't have to be the direct parent of a value!Possible attributes:self.attrs.(any attribute): attributes from the HTML tagself.text: inner textself.element: BeautifulSoup elementFilteringTo select an element with HTML, just write the HTML element.For example, writing<div class='foo'>will select anydivs with classfoo.To filter any attribute, use "filter.*" as an attribute in the element.filter.index=N: this element must be at select(element)[N]filter.regex.*=REGEX: this attribute must match a certain regex. Examples: filter.regex.text=.+, filter.regex.attrs.data=\d+.filter.function=*: you define a function for us, passed as a keyword argument during the constructor. Then, we pass a dict containing attributes ('text', 'element', 'index', and another dictionary called 'attrs'), and your function returns False if the node should be rejected.Post processingIn any tag, you can add the attribute "after" to run on any<list>, <dict>, or <value>'s value.For example,<div id=number>
    <value name=number after=int>self.text</value>
</div>Will call the user-defined functioninton the value returned from self.text.
This applies to any node in the XML tree, including HTML elements.
You can also call thisafterin<list>and<value>tags.NOTE: in lists, this function will be called on the entire list, NOT on individual elements!To define the 'int' function, pass it in the constructor as Parsa((structure), int=function)Something that might be useful would be to have a function calleddf, that makes a pandas dataframe from a list element.Default postprocessing functions:(User-defined functions).<...>: runs type(value).<...>(value). Essentially value.<...>(). Example: ".strip" -> x.strip()Built-in functions like int, float, str, list, dict, etc. Any attribute of the module builtins.Other postprocessing functions:remove_commas: x.replace(",", "")split_commas: x.split(",")split: x.split(" ")You can use function composition by adding a "+" between function names.
For example:
remove_commas+int: "1,000,000" -> "1000000" -> 1000000.If you want to use more than one argument, I suggest writing a wrapper function or making apartialwithfunctools.partial.Required contentBy default, all selectors must exist for a datapoint to be stored.
However, if you want a datapoint to be optional, wrap the selector in<unrequired>.Examplewashington-post.xmlHopefully this explains enough about how it works!
THIS GETS WASHINGTON POST HEADLINES<list name=headlines> // stores children as dicts in a list called 'headlines'
    <div filter.level="any"> // filter.level="any" means they don't have to be direct children
        <div class="headline" filter.level="any"> // finds divs with class headline
            <a> // finds a link
                <value name=link>self.attrs.href</value> // stores the link's href attribute to 'link'
                <value name=headline>self.text</value> // stores the link's text to 'text'. other possibility is 'element', which stores the BS4 node.
            </a>
        </div>
        <span class="author" filter.level="any"> // finds spans with class author
            <a filter.level="any"> // finds any link
                <value name=author>self.text</value> // stores the text to 'author'
            </a>
        </span>
        <div class="art" filter.level="any"> // finds divs with class art
            <p_img filter.level="any"> // img doesn't let you put stuff inside, so it's called p_img
                <value name=image_url>self.attrs.data-hi-res-src</value> // stores an attribute to 'image_url'
            </p_img>
        </div>
    </div>
</list>washington-post.pyimport webparsa
import requests

parser = webparsa.Parsa(washington-post-xml-text)

website_content = requests.get("http://washingtonpost.com")

for headline in parser.parse(website_content)['headlines']:
    print(headline['headline'], headline['author'], headline['image_url'])LicenseStandard MIT license.