Package: screp
======================================**screp**, easy command-line scrapping======================================What is screp?==============**screp** is a command line utility that provides easy and flexible scrappingof HTML documents. It works by finding a set of *anchors* (specified using aCSS selector) and then extracting information relative to those anchors,optionally post processing it using a set of standard operations. For eachanchor it outputs a record formatted according to one of the supported formats(CSV, JSON or general).Invoking screp==============**screp** is invoked using the following syntax::$ screp [OPTION] FORMAT_SPEC PRIMARY_SELECTOR [FILES]where:* FORMAT_SPEC is a format specification, one of:- *-c CSV_FORMAT_SPEC*, formats each record as a comma-separated-values row- *-j JSON_FORMAT_SPEC*, formats each record as a JSON object and the wholeoutput as a list of JSON objects- *-f GENERAL_FORMAT_SPEC*, formats each record according to a general formatwhere computed values are substituted to their specifications (similar tobash parameter substitution)* PRIMARY_SELECTOR is a CSS selector that specifies the *primary anchor*, asdetailed below* FILE can be either a local file or an absolute URL; if no FILEs are specifiedthe standard input is readHow does screp work?====================**screp** tries to automate many of the steps taken when writing your ownscrapper, steps like:* fetching the HTML documents, if necessary* parsing HTML* locating areas of interest in the DOM of the document* locating interesting information around those areas* simple processing of these pieces of information* formatting of the information* outputting the informationTo use screp, you need to take a series of steps:* tell screp where to take the HTML documents; it works with multipledocuments, from sources such as the web, the local file-system or STDIN* define the *primary anchor* using a CSS selector: these are elements throughwhich you access records of interest in the HTML documents* specify the output format; this implies specifying:- *terms*, which are string computed relative to the anchors- how these terms are combined to produce a record; currently screp supportsthree methods of specifying formats:- CSV- JSON- general format* optionally, you can also define *secondary anchors*, which are elementscomputed relative to the *primary anchor* that can be used to define *terms*in a more succinct wayDefining terms==============A *term* has the following format::anchor.accessor.accessor.accessor|filter|filter|filterIn other words, a term is an anchor(primary or secondary) followed by zero ormore accessors followed by zero or more filters.*Accessors* and *filters* (also collectively called *actions*) are functionsthat take the output value of the last function (or the anchor, if this is thefirst action) and output another value. In other words, they form a pipeline.Accessors act on DOM elements and sets (actually ordered lists) of elements,whereas filters act on strings. Each action has an in_type and an out_type. Fora term to be correctly defined the out_type of an action needs to match thein_type of the following action.The supported types are: 'string', 'element', 'element_set'.Actions can have zero or more parameters. When the action takes parameters itis specified as a function::action(parameter1, parameter2, parameter3)When not, only the action name is specified (no parentheses).Finally, terms have restrictions of the out_type of their last action (alsocalled the out_type of the term):* if a term is used inside a format specification, its out_type must be'string'* if a term is used to define a secondary anchor, its out_type must be'element'Examples of terms-----------------These are correct term definitions::'$.parent.parent.attr(title)|upper' outputs 'string''@.desc(".record").first' outputs 'element'anchor.ancestors(".box").children(".price")' outputs 'element_set'Predefined anchors and actions==============================The following anchors are predefined:* **$** is the primary anchor defined by the primary anchor selector* **@** is the primary anchor representing the root of the current documentThe following accessors are predefined:* **first** [in_type='element_set', out_type='element']: returns the firstelement in an element_set* **last** [in_type='element_set', out_type='element']: returns the lastelement in an element_set* **nth(n)** [in_type='element_set', out_type='element']: returns the n-thelement in an element_set; it also supports negative indexes, where -1represents the last element, -2 the second-to-last element, and so on* **class** [in_type='element', out_type='string']: returns the value of the'class' attribute * **id** [in_type='element', out_type='string']: returnsthe value of the 'id' attribute * **parent** [in_type='element',out_type='element']: returns the parent of the current element* **text** [in_type='element', out_type='string']: returns the text enclosed bythe current element* **tag** [in_type='element', out_type='string']: returns the tag of thecurrent element* **attr(attr_name)** [in_type='element', out_type='string']: returns the valueof the current element's attribute with name 'attr_name'* **desc(css_sel)** [in_type='element', out_type='element_set']: returns theordered list of descendants of the current element selected by the CSSselector specified by 'css_sel'* **fdesc(css_sel)** [in_type='element', out_type='element']: equivalent to.desc(css_sel).first* **ancestors(css_sel)** [in_type='element', out_type='element_set']: returnsthe list of ancestors of the current element that satisfy the CSS selectorspecified by 'css_sel'* **children(css_sel)** [in_type='element', out_type='element_set']: returnsthe list of children of the current element that satisfy the CSS selectorspecified by 'css_sel'* **psiblings(css_sel)** [in_type='element', out_type='element_set']: returnsthe list of preceding siblings of the current element that satisfy the CSSselector specified by 'css_sel'* **fsiblings(css_sel)** [in_type='element', out_type='element_set']: returnsthe list of following siblings of the current element that satisfy the CSSselector specified by 'css_sel'* **siblings(css_sel)** [in_type='element', out_type='element_set']: returnsthe list of siblings of the current element that satisfy the CSS selectorspecified by 'css_sel'* **matching(css_sel)** [in_type='element_set', out_type='element_set']:filters an element_set and returns all elements that match the CSS selectorspecified by 'css_sel'The following filters are predefined:* **upper** [in_type='string', out_type='string']: converts string to uppercase* **lower** [in_type='string', out_type='string']: converts string to lowercase* **trim** [in_type='string', out_type='string']: removes spaces at thebeginning and end of the string* **strip(chars)** [in_type='string', out_type='string']: removes charactersspecified by 'chars' at the beginning and end of the string* **replace(old, new)** [in_type='string', out_type='string']: replaces alloccurrences of 'old' with 'new'* **resub(pattern, repl)** [in_type='string', out_type='string']: performs aregular expression substitution; *pattern* and *repl* are have the formatstaken by the **re.sub** Python function from the standard Python library;Specifying output formats=========================CSV format----------The CSV output format is specified using the -c option. Optionally, using the-H option you can specify a CSV header to output before outputting records.Example::-c '$.attr(title), $.parent.desc(".price").text | trim' -H 'name, price'JSON format-----------The JSON output format is defined using the -j option. It formats the output asa JSON list of objects, one for each record. The *--indent-json* flat tellsscrep to indent each object. The format is specified as a comma-separated listof *key=value* pairs, where the *key* represents the JSON key in the recordobject while *value* is a term specification.Example::- j 'text=$.text, ptext=$.parent.text | upper, gptext=$.parent.parent.text'General format--------------Then general format is specified by a general string containing termspecifications. To distinguish it from the general format, each termspecification is surrounded by braces. When formatting a record each termspecification is substituted with the computed value for that term.Example::-f 'some header {$.parent.text | replace("X", "Y")} some middle {$.tag} sometail'Specifying secondary anchors============================Secondary anchors are specified using the -a option. There can be any number ofsecondary anchors definitions. The definitions have the format**<name>=<term>** where <name> is an identifier and <term> is a term definitionrelative to any of the previously defined anchors (primary or secondary) thathas outputs an element. Secondary anchors can be redefined in later -a optionsbut only the last definition is retained.Secondary anchors examples--------------------------These are examples of secondary anchors definitions::-a 'p=$.parent' -a 'gp=p.parent'-a 'interesting=$.fdesc(".interesting-class")' -a'interesting=interesting.parent'