Package: xextract
Extract structured data from HTML and XML documents like a boss.xextractis simple enough for writing a one-line parser, yet powerful enough to be used in a big project.FeaturesParsing of HTML and XML documentsSupportsxpathandcssselectorsSimple declarative style of parsersBuilt-in self-validation to let you know when the structure of the website has changedSpeed - under the hood the library useslxml librarywith compiled xpath selectorsTable of ContentsA little taste of itInstallationParsersStringUrlDateTimeDateElementGroupPrefixParser parametersnamecss / xpathquantattrcallbackchildrennamespacesHTML vs. XML parsingA little taste of itLet’s parseThe Shawshank Redemption’s IMDB page:# fetch the website>>>importrequests>>>response=requests.get('http://www.imdb.com/title/tt0111161/')# parse like a boss>>>fromxextractimportString,Group# extract title with css selector>>>String(css='h1[itemprop="name"]',quant=1).parse(response.text)u'The Shawshank Redemption'# extract release year with xpath selector>>>String(xpath='//*[@id="titleYear"]/a',quant=1,callback=int).parse(response.text)1994# extract structured data>>>Group(css='.cast_list tr:not(:first-child)',children=[...String(name='name',css='[itemprop="actor"]',attr='_all_text',quant=1),...String(name='character',css='.character',attr='_all_text',quant=1)...]).parse(response.text)[{'name':u'Tim Robbins','character':u'Andy Dufresne'},{'name':u'Morgan Freeman','character':u"Ellis Boyd 'Red' Redding"},...]InstallationTo installxextract, simply run:$pipinstallxextractRequirements: six, lxml, cssselectSupported Python versions are 2.6, 2.7, 3.x.Windows users can download lxml binaryhere.ParsersStringParameters:name(optional),css / xpath(optional, default"self::*"),quant(optional, default"*"),attr(optional, default"_text"),callback(optional),namespaces(optional)Extract string data from the matched element(s).
Extracted value is always unicode.By default,Stringextracts the text content of only the matched element, but not its descendants.
To extract and concatenate the text out of every descendant element, useattrparameter with the special value"_all_text":Useattrparameter to extract the data from an HTML/XML attribute.Usecallbackparameter to post-process extracted values.Example:>>>fromxextractimportString>>>String(css='span',quant=1).parse('<span>Hello <b>world</b>!</span>')u'Hello !'>>>String(css='span',quant=1,attr='class').parse('<span class="text-success"></span>')u'text-success'# use special `attr` value `_all_text` to extract and concantenate text out of all descendants>>>String(css='span',quant=1,attr='_all_text').parse('<span>Hello <b>world</b>!</span>')u'Hello world!'# use special `attr` value `_name` to extract tag name of the matched element>>>String(css='span',quant=1,attr='_name').parse('<span>hello</span>')u'span'>>>String(css='span',callback=int).parse('<span>1</span><span>2</span>')[1,2]UrlParameters:name(optional),css / xpath(optional, default"self::*"),quant(optional, default"*"),attr(optional, default"href"),callback(optional),namespaces(optional)Behaves likeStringparser, but with two exceptions:default value forattrparameter is"href"if you passurlparameter toparse()method, the absolute url will be constructed and returnedIfcallbackis specified, it is calledafterthe absolute urls are constructed.Example:>>>fromxextractimportUrl,Prefix>>>content='<div id="main"> <a href="/test">Link</a> </div>'>>>Url(css='a',quant=1).parse(content)u'/test'>>>Url(css='a',quant=1).parse(content,url='http://github.com/Mimino666')u'http://github.com/test'# absolute url address. Told ya!>>>Prefix(css='#main',children=[...Url(css='a',quant=1)...]).parse(content,url='http://github.com/Mimino666')# you can pass url also to ancestor's parse(). It will propagate down.u'http://github.com/test'DateTimeParameters:name(optional),css / xpath(optional, default"self::*"),format(required),quant(optional, default"*"),attr(optional, default"_text"),callback(optional)namespaces(optional)Returns thedatetime.datetimeobject constructed out of the extracted data:datetime.strptime(extracted_data, format).formatsyntax is described in thePython documentation.Ifcallbackis specified, it is calledafterthe datetime objects are constructed.Example:>>>fromxextractimportDateTime>>>DateTime(css='span',quant=1,format='%d.%m.%Y %H:%M').parse('<span>24.12.2015 5:30</span>')datetime.datetime(2015,12,24,50,30)DateParameters:name(optional),css / xpath(optional, default"self::*"),format(required),quant(optional, default"*"),attr(optional, default"_text"),callback(optional)namespaces(optional)Returns thedatetime.dateobject constructed out of the extracted data:datetime.strptime(extracted_data,format).date().formatsyntax is described in thePython documentation.Ifcallbackis specified, it is calledafterthe datetime objects are constructed.Example:>>>fromxextractimportDate>>>Date(css='span',quant=1,format='%d.%m.%Y').parse('<span>24.12.2015</span>')datetime.date(2015,12,24)ElementParameters:name(optional),css / xpath(optional, default"self::*"),quant(optional, default"*"),callback(optional),namespaces(optional)Returns lxml instance (lxml.etree._Element) of the matched element(s).
If you use xpath expression and match the text content of the element (e.g.text()or@attr), unicode is returned.Ifcallbackis specified, it is called withlxml.etree._Elementinstance.Example:>>>fromxextractimportElement>>>Element(css='span',quant=1).parse('<span>Hello</span>')<Elementspanat0x2ac2990>>>>Element(css='span',quant=1,callback=lambdael:el.text).parse('<span>Hello</span>')u'Hello'# same as above>>>Element(xpath='//span/text()',quant=1).parse('<span>Hello</span>')u'Hello'GroupParameters:name(optional),css / xpath(optional, default"self::*"),children(required),quant(optional, default"*"),callback(optional),namespaces(optional)For each element matched by css/xpath selector returns the dictionary containing the data extracted by the parsers listed inchildrenparameter.
All parsers listed inchildrenparametermusthavenamespecified - this is then used as the key in dictionary.Typical use case for this parser is when you want to parse structured data, e.g. list of user profiles, where each profile contains fields like name, address, etc. UseGroupparser to group the fields of each user profile together.Ifcallbackis specified, it is called with the dictionary of parsed children values.Example:>>>fromxextractimportGroup>>>content='<ul><li id="id1">michal</li> <li id="id2">peter</li></ul>'>>>Group(css='li',quant=2,children=[...String(name='id',xpath='self::*',quant=1,attr='id'),...String(name='name',xpath='self::*',quant=1)...]).parse(content)[{'name':u'michal','id':u'id1'},{'name':u'peter','id':u'id2'}]PrefixParameters:css / xpath(optional, default"self::*"),children(required),namespaces(optional)This parser doesn’t actually parse any data on its own. Instead you can use it, when many of your parsers share the same css/xpath selector prefix.Prefixparser always returns a single dictionary containing the data extracted by the parsers listed inchildrenparameter.
All parsers listed inchildrenparametermusthavenamespecified - this is then used as the key in dictionary.Example:# instead of...>>>String(css='#main .name').parse(...)>>>String(css='#main .date').parse(...)# ...you can use>>>fromxextractimportPrefix>>>Prefix(css='#main',children=[...String(name="name",css='.name'),...String(name="date",css='.date')...]).parse(...)Parser parametersnameParsers:String,Url,DateTime,Date,Element,GroupDefault value:NoneIf specified, then the extracted data will be returned in a dictionary, with thenameas the key and the data as the value.All parsers listed inchildrenparameter ofGrouporPrefixparsermusthavenamespecified.
If multiple children parsers have the samename, the behavior is undefined.Example:# when `name` is not specified, raw value is returned>>>String(css='span',quant=1).parse('<span>Hello!</span>')u'Hello!'# when `name` is specified, dictionary is returned with `name` as the key>>>String(name='message',css='span',quant=1).parse('<span>Hello!</span>'){'message':u'Hello!'}css / xpathParsers:String,Url,DateTime,Date,Element,Group,PrefixDefault value (xpath):"self::*"Use eithercssorxpathparameter (but not both) to select the elements from which to extract the data.Under the hood css selectors are translated into equivalent xpath selectors.For the children ofPrefixorGroupparsers, the elements are selected relative to the elements matched by the parent parser.Example:Prefix(xpath='//*[@id="profile"]',children=[# equivalent to: //*[@id="profile"]/descendant-or-self::*[@class="name"]String(name='name',css='.name',quant=1),# equivalent to: //*[@id="profile"]/*[@class="title"]String(name='title',xpath='*[@class="title"]',quant=1),# equivalent to: //*[@class="subtitle"]String(name='subtitle',xpath='//*[@class="subtitle"]',quant=1)])quantParsers:String,Url,DateTime,Date,Element,GroupDefault value:"*"quantspecifies the expected number of elements to be matched with css/xpath selector. It serves two purposes:Number of matched elements is checked against thequantparameter. If the number of elements doesn’t match the expected quantity,xextract.parsers.ParsingErrorexception is raised. This way you will be notified, when the website has changed its structure.It tells the parser whether to return a single extracted value or a list of values. See the table below.Syntax forquantmimics the regular expressions.
You can either pass the value as a string, single integer or tuple of two integers.Depending on the value ofquant, the parser returns either a single extracted value or a list of values.Value ofquantMeaningExtracted data"*"(default)Zero or more elements.List of values"+"One or more elements.List of values"?"Zero or one element.Single value orNonenumExactlynumelements.You can pass either string or integer.num== 0:Nonenum== 1: Single valuenum> 1: List of values(num1, num2)Number of elements has to be betweennum1andnum2, inclusive.You can pass either a string or 2-tuple.List of valuesExample:>>>String(css='.full-name',quant=1).parse(content)# return single valueu'John Rambo'>>>String(css='.full-name',quant='1').parse(content)# same as aboveu'John Rambo'>>>String(css='.full-name',quant=(1,2)).parse(content)# return list of values[u'John Rambo']>>>String(css='.full-name',quant='1,2').parse(content)# same as above[u'John Rambo']>>>String(css='.middle-name',quant='?').parse(content)# return single value or NoneNone>>>String(css='.job-titles',quant='+').parse(content)# return list of values[u'President',u'US Senator',u'State Senator',u'Senior Lecturer in Law']>>>String(css='.friends',quant='*').parse(content)# return possibly empty list of values[]>>>String(css='.friends',quant='+').parse(content)# raise exception, when no elements are matchedxextract.parsers.ParsingError:ParserStringmatched0elements("+"expected).attrParsers:String,Url,DateTime,DateDefault value:"href"forUrlparser."_text"otherwise.Useattrparameter to specify what data to extract from the matched element.Value ofattrMeaning"_text"Extract the text content of the matched element."_all_text"Extract and concatenate the text content of
the matched element and all its descendants."_name"Extract tag name of the matched element.att_nameExtract the value out ofatt_nameattribute of
the matched element.If such attribute doesn’t exist, empty string is
returned.Example:>>>fromxextractimportString,Url>>>content='<span class="name">Barack <strong>Obama</strong> III.</span> <a href="/test">Link</a>'>>>String(css='.name',quant=1).parse(content)# default attr is "_text"u'Barack  III.'>>>String(css='.name',quant=1,attr='_text').parse(content)# same as aboveu'Barack  III.'>>>String(css='.name',quant=1,attr='_all_text').parse(content)# all textu'Barack Obama III.'>>>String(css='.name',quant=1,attr='_name').parse(content)# tag nameu'span'>>>Url(css='a',quant='1').parse(content)# Url extracts href by defaultu'/test'>>>String(css='a',quant='1',attr='id').parse(content)# non-existent attributes return empty stringu''callbackParsers:String,Url,DateTime,Date,Element,GroupProvides an easy way to post-process extracted values.
It should be a function that takes a single argument, the extracted value, and returns the postprocessed value.Example:>>>String(css='span',callback=int).parse('<span>1</span><span>2</span>')[1,2]>>>Element(css='span',quant=1,callback=lambdael:el.text).parse('<span>Hello</span>')u'Hello'childrenParsers:Group,PrefixSpecifies the children parsers for theGroupandPrefixparsers.
All parsers listed inchildrenparametermusthavenamespecifiedCss/xpath selectors in the children parsers are relative to the selectors specified in the parent parser.Example:Prefix(xpath='//*[@id="profile"]',children=[# equivalent to: //*[@id="profile"]/descendant-or-self::*[@class="name"]String(name='name',css='.name',quant=1),# equivalent to: //*[@id="profile"]/*[@class="title"]String(name='title',xpath='*[@class="title"]',quant=1),# equivalent to: //*[@class="subtitle"]String(name='subtitle',xpath='//*[@class="subtitle"]',quant=1)])namespacesParsers:String,Url,DateTime,Date,Element,Group,PrefixWhen parsing XML documents containing namespace prefixes, pass the dictionary mapping namespace prefixes to namespace URIs.
Use then full name for elements in xpath selector in the form"prefix:element"As for the moment, youcannot use default namespacefor parsing (seelxml docsfor more information).  Just use an arbitrary prefix.Example:>>>content='''<?xml version='1.0' encoding='UTF-8'?>
... <movie xmlns="http://imdb.com/ns/">
...   <title>The Shawshank Redemption</title>
...   <year>1994</year>
... </movie>'''>>>nsmap={'imdb':'http://imdb.com/ns/'}# use arbitrary prefix for default namespace>>>Prefix(xpath='//imdb:movie',namespaces=nsmap,children=[# pass namespaces to the outermost parser...String(name='title',xpath='imdb:title',quant=1),...String(name='year',xpath='imdb:year',quant=1)...]).parse(content){'title':u'The Shawshank Redemption','year':u'1994'}HTML vs. XML parsingTo extract data from HTML or XML document, simply callparse()method of the parser:>>>fromxextractimport*>>>parser=Prefix(...,children=[...])>>>extracted_data=parser.parse(content)contentcan be either string or unicode, containing the content of the document.Under the hoodxextactuses eitherlxml.etree.XMLParserorlxml.etree.HTMLParserto parse the document.
To select the parser,xextractlooks for"<?xml"string in the first 128 bytes of the document. If it is found, thenXMLParseris used.To force either of the parsers, you can callparse_html()orparse_xml()method:>>>parser.parse_html(content)# force lxml.etree.HTMLParser>>>parser.parse_xml(content)# force lxml.etree.XMLParser