Package: take
A DSL for extracting data from a web page. The DSL serves two purposes:
finds elements and extracts their text or attribute values. The main
reason for developing this is to have all the CSS selectors for scraping
a site in one place (I prefer CSS selectors over anything else).The DSL wrapsPyQuery.A few links:Github repositoryPyPi packageDiscussion groupExampleGiven the following take template:$ h1 | text
    save: h1_title
$ ul
    save each: uls
        $ li
            | 0 [title]
                save: title
            | 1 text
                save: second_li
$ p | 1 text
    save: p_textAnd the following HTML:<div><h1>Le Title 1</h1><p>Some body here</p><p>The second body here</p><ulid="a"><lititle="a less than awesome title">A first li</li><li>Second li in list #a</li><li>A third li</li></ul><ulid="b"><lititle="some awesome title">B first li</li><li>Second li in list #b</li><li>B third li</li></ul></div>The following data will be extracted (presented in JSON format):{"h1_title":"Le Title 1","p_text":"The second body here","uls":[{"title":"a less than awesome title","second_li":"Second li in list #a"},{"title":"some awesome title","second_li":"Second li in list #b"}]}Take templates always result in a single pythondict.The template can also be written in the following, more concise, syntax:$ h1 | text ;                   : h1_title
$ ul
    save each                   : uls
        $ li
            | 0 [title] ;           : title
            | 1 text ;              : second_li
$ p | 1 text ;                  : p_textThe example above is formatted with extra whitespace to make the structure
of the resulting data more apparent.More ExamplesFor more complex examples:Scraping thereddit home pageInline versionVerbose versionScraping the latestweb-scraping questionson Stack Overflow:Overviewquestions-listing.takequestion-page.takeInstallpip install takeUsageCreating a Take TemplateA take template can be created from a file via the static methodTakeTemplate.from_file().fromtakeimportTakeTemplatett=TakeTemplate.from_file('yourfile.take')TheTakeTemplateconstructor can be used to create a template from either
abasestringor anIterable.To create a template from a string:fromtakeimportTakeTemplateTMPL="""
$ nav a
    save each: nav
        | text
            save: text
        | [href]
            save: link
"""tt=TakeTemplate(TMPL)Additionally, abase_urlkeyword argument can be specified which
will cause relative URLs to be made absolute via the value of thebase_urlparameter for any documents that are processed.tt=TakeTemplate.from_file('yourfile.take',base_url='http://www.example.com')tt=TakeTempalte(TMPL,base_url='http://www.example.com')If abase_urlis provided when the template is used, it will
override thebase_urlprovided when the template was created. Thebase_urlparameter must be provided as a keyword argument.Using a Take TemplateTo parse from a URL:data=tt(url='http://www.example.com')To parse from a html string:data=tt('<div>hello world</div>')To parse from a file:data=tt(filename=path_to_html_file)Alternatively, thetake()method can be used:data=tt.take(url='http://www.example.com')Valid parameters for the template callable or thetake()method are
the same as those for thePyQuery constructor.Additionally, if the'base_url'keyword parameter is supplied, all
relative URLs will be made absolute via the value of'base_url'.data=tt(url='http://www.example.com',base_url='http://www.example.com')Take TemplatesTake templates are whitespace sensitive and are comprised of three types
of statements:Comment Lines# some commentQueries$ h1| text$ h1 | 0 textDirectivessave: h1_titlesave each: commentsmerge: *def: get commentsComment LinesAny line with a#as the first non-whitespace character is considered a comment line.# this line is a comment
# the third line is a CSS selector query
$ #main-nav aComment lines are completely ignored. Partially commented lines and multi-line comments are not supported at this time.QueriesThere are two main types of queries in take templates:CSS selector queriesNon-CSS selector queriesThe reason they’re divided like this is because CSS selectors always go
first on the line and they can be followed by non-CSS selector queries.
Non-CSS selector queries can’t be followed by CSS selector queries.
Seems easier to read this way, but it’s arbitrary and may change.CSS Selector QueriesCSS selector queries start with$and end either at the end of the
line, the|character or the;character. The|character
is the starting character for non-CSS selector queries, and the;character ends the statement and starts aninline sub-context.$ #siteTable .thing | text$ .domain aIn the first example above, the CSS selector query is#siteTable .thing. The second is.domain a.The CSS selectors are passed toPyQuery, so anything PyQuery can
accept can be used. From what I understand, there are a fewbugsin
PyQuery (that may be in the underlying librarieslxmlorcssselect). Those will come up.Non-CSS Selector QueriesNon-CSS selector queries start with|and continue until the;character or the
line ends. There are five non-CSS selector queries:Element indexesSyntax: an integer| 0will return the first element in the current context| 1will return the second element in the current context|-1will return the last element in the current contextAttribute retrievalSyntax:[attr]| [href]will return the value of thehrefattribute of the
first element in the current context| 1 [href]will return the value of thehrefattribute of the
second element in the current contextText retrievalSyntax:text| textwill return the text of the current context| 1 textwill first get the second element in the current context
and then return it’s textOwn text retrievalSyntax:own_text| own_textwill return the text of the current context without the text
from its children| 1 own_textwill first get the second element in the current context
and then return it’s text without the text from its childrenField retrievalSyntax:.field_name| .descriptionwill do a dictionary lookup on the context and retrieve
the value of the'description'item| .parent.childwill do a dictionary lookup on the context and retrieve
the value of the'parent'and then it will lookup'child'on that valueOrder matters: Index queries should precede other queries. Also, only one
of[attr],text,own_textor.field_namequeries can be used.IndentationThe level of indentation on each line defines the context for the line.The root context of a take template is the current document being
processed. Every statement that is not indented is executed against the
document being processed.Each line that is indented more deeply has a context that is the result
of the last query in the parent context. For example:$ #some-id
    $ li
    $ divThe query on the first line is executed against the document being
processed. The query on the second line is executed against the result
of the first line. So, the second line is synonomous with$#some-idli. The query on the third line is also executed against
the result of the first line. So, it can be re-written as$#some-iddiv.Another example:$ a
    | 0
        | text
        | [href]The third and fourth lines retrieve the text and href attribute,
respectively, from the first<a>in the document being processed.
This could be rewritten as:$ a | 0
    | text
    | [href]Inline Sub ContextsInline sub-contexts allow multuple statements per line. The syntax is:statement ; sub-context-statementThe main thing to note is: whatever comes after the semi-colin is treated as if it were a line with deeper indentation.Inline sub-contexts are primarily used with directives. For example, the following take template:$ h1 | 0 text
    save: section_titleCan be re-written as:$ h1 | 0 text ; save: document_titleBoth templates save the text in the first<h1>element into the resultdictwith the key'document_title'. More onsave directiveslater.DirectivesDirectives are commands that are executed against the current context.
They’re format is a directive name followed by an optional parameter list:<directive_name> [: <parameter>[<whitespace or comma> <parameter>]*]?An example of asavedirective:save : some_nameNot all directives require parameters. For example, theshrinkdirective,
which collapses whitespace, does not:shrinkThe following directives are built-in:save, alias:Saves a value.save eachCreates a list of results.namespace, alias+Creates childdictfor saving values into.shrinkCollapses and trims whitespace.defDefines a new directive.Currently only new directives defined in the current document are available.merge, alias>>Copies a value from a directive’s result into the template’s result.Save DirectiveAlias::Save directives save the context into the resultdict. These are
generally only intended to be applied to the result of non-CSS Selector
queries.The syntax is:save: <identifier>:is an alias forsave:. So, a save directive can also be written as:: <identifier>The identifier can contain anything except whitespace, a comma (,) or a semi-colin (;).
Also, the identifier can contain dots (.), which designate sub-dictsfor
saving.For example, the following take template:$ a | 0
    | text
        save: first_a.description
    | [href]
        save: first_a.urlAnd the following HTML:<div><ahref="http://www.example.com">fo sho</a><ahref="http://www.another.com">psych out</a></div>Will result in the following pythondict:{'first_a':{'description':'fo sho','url':'http://www.example.com'}}Using the:alias, the template can be written as:$ a | 0
    | text
        : first_a.text
    | [href]
        : first_a.hrefOr, more succinctly:$ a | 0
    | text ;        : first_a.text
    | [href] ;      : first_a.hrefSave Each DirectiveSave each directives produce adictfor each element in the context. Generally, these are used for repeating elements on a page. In thereddit sample, a save each directive is used to save each of the reddit entries.The syntax is:save each: <identifier>
    <sub-context>The identifier can contain anything except whitespace, a comma (,) or a semi-colin (;).
Also, the identifier can contain dots (.), which designate sub-dicts for
saving.Save each directives apply the next sub-context to each of the elements
of their context value. Put another way, save each directives repeatedly
process each element of their context.For example, in the following take template, the| textand| [href]queries (along with saving the results) will be applied to
every<a>in the document.$ a
    save each: anchors
        | text
            save: description
        | [href]
            save: urlApplying the above take template to the following HTML:<div><ahref="http://www.example.com">fo sho</a><ahref="http://www.another.com">psych out</a></div>Will result in the following pythondict:{'anchors':[{'description':'fo sho','url':'http://www.example.com'},{'description':'psych out','url':'http://www.another.com'}]}Namespace DirectiveAlias:+Namespace directives create a sub-dicton the current result-value and everyting in the
next sub-context is saved into the newdict.The syntax is:namespace: <identifier>
    <sub-context><identifier>is the key the sub-dictis saved as.An example:$ a | 0
    namespace: first_a
        | text
            save: description
        | [href]
            save: urlApplying the above take template to the following HTML:<div><ahref="http://www.example.com">fo sho</a><ahref="http://www.another.com">psych out</a></div>Will result in the following pythondict:{'first_a':{'description':'fo sho','url':'http://www.example.com'}}Thedescriptionandurlfields are saved in thefirst_anamespace. This reduces
the need for save directives like:first_a.description.+is an alias for thenamespacedirective. So, the template above can also be written as:$ a | 0
    +       : first_a
        | text
            save: description
        | [href]
            save: urlOr, more succinctly, using inline sub-contexts and the:alias for save:$ a | 0 ; +         : first_a
        | text ;        : description
        | [href] ;      : urlShrink DirectiveTheshrinkdirective trims and collapses whitespace from text. It doesn’t take any parameters,
so the usage is just the wordshrink:$ p | text ;            : with_spacing
$ p | text ; shrink ;   : shrink_on_textIf applied to an element, it will be applied to the element’s text.$ p ; shrink ;          : shrink_on_elemApplying the above statements to the following HTML:<p>Hello       World!</p>Will result in the following pythondict:{'with_spacing':'Hello       World!','shrink_on_text':'Hello World!','shrink_on_elem':'Hello World!'}Def DirectiveThedefdirective saves a sub-context as a custom directive which can be invoked later. This is a
way to re-use sections of a take template. Directives created in this fashionalways result in a newdict.The syntax is:def: <identifier>
    <sub-context>For example:def: get first link
    $ a | 0
        | text ;    : description
        | [href] ;  : urlIn the above template, a new directive namedget first linkis created. The new directive saves
the text and href attribute from the first<a>element in the context onto which it is
invoked. The directive will always result in a newdictcontainingdescriptionandurlkeys.The identifier can contain spaces; all spaces are collapsed to be a single space,
e.g.def: some    nameis collapsed todef: some name.Directives created bydefare invoked without parameters.The example below defines a custom directive and applies it against the first<nav>element and the first<aside>element.def: get first link
    $ a | 0
        | text ;    : description
        | [href] ;  : url

$ nav
    get first link
        save: first_nav_link
$ aside
    get first link
        save: first_aside_linkGiven the following HTML:<div><nav><ahref="/local/a">nav item A</a><ahref="/local/b">nav item B</a></nav><aside><p>some description</p><ahref="http://ext.com/a">aside item A</a><ahref="http://ext.com/b">aside item B</a></aside></div>The template would result in:{'first_nav_link':{'description':'nav item A','url':'/local/a'},'first_aside_link':{'description':'aside item A','url':'http://ext.com/a'}}Each time the directive is invoked it returns a pythondictcontaining'description'and'url'keys. The return value of the first invocation is saved into the template’s result as'first_nav_link'. The second return value is saved as'first_aside_link'Another way to save the data from a custom directive is to use the| .propertyquery. This allows renaming, too:def: get first link
    $ a | 0
        | text ;    : description
        | [href] ;  : url

$ nav
    get first link
        | .url ;
            save: first_nav_url
$ aside
    get first link
        | .url ;
            save: first_aside_urlThe above template would result in the followingdict:{'first_nav_url':'/local/a','first_aside_url':'http://ext.com/a'}Merge DirectiveAlias:>>Themergedirective copies properties from the context’s value and saves them into the result value. The main
use-case is extracting data from the result of a custom directive.mergeperforms a shallow copy.The syntax is:merge: <field> [<field>]*The parameter(s) are the keys to copy. They are separated by spaces or a comma and new line.The special parameter*can be used to copy all the keys. If used, it should be the only parameter:merge: *Note:mergeexpects the context’s value to be adict; behind the scenes it uses themapping[key]syntax.An example:def: link info
    | text              : text
    | [href]            : url
    | [title]           : title

$ footer a
    save each               : footer_links
        link info
            merge               : urlApplying the above take template to the following HTML:<html><head>...</head><body><divclass="main">...</div><footer><ul><li><ahref="/about"title="All about our company">Team</a></li><li><ahref="https://blog.example.com"title="Our self-promos">Blog</a></li><li><ahref="www.facebook.com/example"title="Our facebook page">Facebook</a></li><li><ahref="/privacy"title="Legalese">Privacy</a></li></ul></footer></body></html>Will result in the following pythondict:{'footer_links':[{'url':'/about'},{'url':'https://blog.example.com'},{'url':'www.facebook.com/example'},{'url':'/privacy'}]}To copy more than one property, separate the property names with a space or a comma and new-line:# separated by spaces
merge               : url title

                    # separated with comma line-continuation
merge               : url,
                      title

                    # using the `>>` alias
>>                  : url,
                      title