Package: pyboro
# What is pyboro? #pyboro is a regular expression lexer built with python.pyboro is suitable for parsing syntax and building symbol tables,and it's easy to use this information to extend into building yourown code generation and so on.The name comes from 'borosilicate', which is the type of glass usedin making [Pyrex](http://en.wikipedia.org/wiki/Pyrex) glassware. The repositorywas originally called pyRex, but this conflicted with similarly named packagesalready existing in PyPi.# Installation #pip install pyboroeasy_install pyboroThen:import pyboroOr:from pyboro import Lexerfrom pyboro import Consumer# The Lexer Module #The Lexer module (`pyboro.Lexer`) is used to define your symboltables. You'll create `ParseMap` objects for each of theregular expressions you want to be able to parse.These are set up in the following way:my_parser = pyboro.Lexer.ParseMap((("token name",  "regular expression",   handler),#...))In the above, "token name" is a brief title for the regular expression.For those that will be ignored, the name is irrevant for functionality,but good for people who might have to maintain your code. For tokens that willbe preserved, the token name is how you will access the matched expression.# `let x = 17;`, where the ParseMap searches for a variable name and an assignment:result = my_parser.parse(input_string)print(result['variable name']) #prints 'x'print(result['assigment']) #prints '17'"regular expression" is the regular expression matches the title."handler" is either a function that you create, OR a ParseMap constant.The ParseMap class has two constants:pyboro.Lexer.ParseMap.IGNORE     # consumes the input matching the regex, but does not store itpyboro.Lexer.ParseMap.LITERAL    # consumes and stores the input exactly as its foundIf you use the name of a function for the handler, it must take a single string as an argument,and output a single string. This allows you to transform or verify inputs further. Note that in python,basic types are also functions. This makes it easy to convert the input strings into integers, etc("integer assignment", regex_foo,   int)For a more in-depth look at the Lexer, read through [the tutorial](#tutorial).# The Consumer Module #While the Lexer module is used to set up your symbol tables, the Consumer module is used to actuallyconsume input and return result tables.A Consumer object requires  a single argument a list of ParseMaps:my_consumer = Consumer.Consumer([my_parser,my_other_parser,my_third_parser])Whenever an input string is given to that object, it will iterate over the input, checking againsteach of the ParseMaps in order. It will do this until either a) all of the input is consumed (hooray!)or b) a syntax error is found (boo).It will return the results from each of the ParseMaps, along with a reference to the map that found it.my_consumer = Consumer.Consumer([EmailAddress, Name, FavoriteColor])results = Nonewith open("input_file.txt") as f:results = my_consumer.parse(f.read()) #will throw an error if the entire file cannot be parsed cleanlyfor result in results:if result is EmailAddress:print result["address"]Optional arguments to the conumser module:help: A text to be displayed if an error is foundformatting_func: a function which takes input, formats it, and returns it.This will produce helpful error messages like:"Syntax error near 'dafsdfdfdjfkdljkj 12;', expecting valid type."where 'dafsdfdfdjfkdljkj 12;' is the output of formatting\_func, and"valid type" is the help message.<a name="tutorial"></a># Brief Tutorial of Lexing#Before you begin using pyboro, you should have an idea how to useregular expressions with Python's `re` module. If you need help withthis, check out the [official documentation](http://docs.python.org/library/re.html).Next, you'll need an idea of what you'll be parsing. For this example,let's say that you want to parse this line:int aNumber = 17;We have five things here:+ `int` : a variable declaration+ `aNumber` : an identifier+ `=` : an operator+ `17` : an assignment to the variableWe also have whitespace between these things.What are the regular expressions for these things?+ `int`+ the literal 'int'+ `[ \t]+`+ at least one space or tab+ `[_a-zA-Z][_a-zA-Z0-9]*`+ A string of at least one character. It must start with an underscore or letter, but may have numbers after it.+ `[ \t]*`+ any number of spaces or tabs+ `\=`+ The assignment operator+ `[ \t]*`+ any number of spaces or tabs+ `[^;]+`+ at least one character that isn't a semicolon (more on this in a second)+ `[ \t]*`+ any number of spaces or tabs+ `;`+ a semicolon.The reason we're allowing any number of things (except semicolons) inthe assignment is because we may not have a literal integer here. We mayhave a function call, for instance, which returns an integer. We'll dealwith whatever that is at a later time (not in this example).Now that we've done that, we can create a`ParseMap`To create a `ParseMap` of everything above, we do it like this:```pythona_parser = Lexer.ParseMap((("type_declaration" ,   "int",                      Lexer.ParseMap.LITERAL),("whitespace"       ,   "[ \t]+",                   Lexer.ParseMap.IGNORE),("identifier"       ,   "[_a-zA-Z][_a-zA-Z0-9]*",   Lexer.ParseMap.LITERAL),("whitespace"       ,   "[ \t]*",                   Lexer.ParseMap.IGNORE),("assignment_op"    ,   "\=",                       Lexer.ParseMap.IGNORE),("assignment"       ,   "[^;]+",                    validate_as_integer),("whitespace"       ,   "[ \t]*",                   Lexer.ParseMap.IGNORE),("statement_end"    ,   ";",                        Lexer.ParseMap.IGNORE)))```We give the ParseMap a tuple of tuples, each of which has the following:1. A short description of what the regex represents2. A regular expression3. Something that tells the parser what to do with the token foundLet's examine the third column:1. `Lexer.ParseMap.LITERAL` tells the parseMap to store the data found exactly as it is stated.2. `Lexer.ParseMap.IGNORE` will discard the information3. `validate_as_integer` is a function that maybe we've already written, which takes in thetoken as a string, and returns some other value which will be stored.We can then feed the parser input like so:results = a_parser.parse(input_string)`results` will be an `OrderedDict` whose keys are the descriptions above,and whose values are the parsed values (or literal values). Therefore, in thisinstance, `results` will look like this:```pythonresults['type_declaration'] = 'int'results['identifier'] = 'aNumber'results['assignment'] = 17```