Package: pyleri
Python Left-Right ParserPyleri is an easy-to-use parser created forSiriDB. We first usedlrparsingand wrotejslerifor auto-completion and suggestions in our web console. Later we found small issues within thelrparsingmodule and also had difficulties keeping the language the same in all projects. That is when we decided to create Pyleri which can export a created grammar to JavaScript, C, Python, Go and Java.Gabriele Tomassettiwrote a tutorialabout the pyleri library.Related projectsInstallationQuick usageGrammarGrammar.parse()Grammar.export_js()Grammar.export_c()Grammar.export_go()Grammar.export_java()Grammar.export_py()Resultis_validPositionTreeExpectingElementsKeywordRegexTokenTokensSequenceChoiceRepeatListOptionalRefPrioRelated projectsjsleri: JavaScript parserlibcleri: C parsergoleri: Go parserjleri: Java parserInstallationThe easiest way is to use PyPI:sudo pip3 install pyleriQuick usage# Imports, note that we skip the imports in other examples...frompyleriimport(Grammar,Keyword,Regex,Sequence)# Create a Grammar Class to define your languageclassMyGrammar(Grammar):r_name=Regex('(?:"(?:[^"]*)")+')k_hi=Keyword('hi')START=Sequence(k_hi,r_name)# Compile your grammar by creating an instance of the Grammar Class.my_grammar=MyGrammar()# Use the compiled grammar to parse 'strings'print(my_grammar.parse('hi "Iris"').is_valid)# => Trueprint(my_grammar.parse('bye "Iris"').is_valid)# => Falseprint(my_grammar.parse('bye "Iris"').as_str())# => error at position 0, expecting: hiGrammarWhen writing a grammar you should subclass Grammar. A Grammar expects at least aSTARTproperty so the parser knows where to start parsing. Grammar has some default properties which can be overwritten likeRE_KEYWORDS, which will be explained later. Grammar also has a parse method:parse(), and a few export methods:export_js(),export_c(),export_py(),export_go()andexport_java()which are explained below.parsesyntax:Grammar().parse(string)Theparse()method returns a result object which has the following properties that are further explained inResult:expectingis_validpostreeexport_jssyntax:Grammar().export_js(js_module_name='jsleri',js_template=Grammar.JS_TEMPLATE,js_indent=' '*4)Optional keyword arguments:js_module_name: Name of the JavaScript module. (default: 'jsleri')js_template: Template String used for the export. You might want to look at the default string which can be found at Grammar.JS_TEMPLATE.js_indent: indentation used in the JavaScript file. (default: 4 spaces)For example when using our Quick usage grammar, this is the output when runningmy_grammar.export_js():/* jshint newcap: false *//** This grammar is generated using the Grammar.export_js() method and* should be used with the jsleri JavaScript module.** Source class: MyGrammar* Created at: 2015-11-04 10:06:06*/'use strict';(function(Regex,Sequence,Keyword,Grammar){varr_name=Regex('^(?:"(?:[^"]*)")+');vark_hi=Keyword('hi');varSTART=Sequence(k_hi,r_name);window.MyGrammar=Grammar(START,'^\w+');})(window.jsleri.Regex,window.jsleri.Sequence,window.jsleri.Keyword,window.jsleri.Grammar);export_csyntax:Grammar().export_c(target=Grammar.C_TARGET,c_indent=' '*4)Optional keyword arguments:target: Name of the c module. (default: 'grammar')c_indent: indentation used in the c files. (default: 4 spaces)The return value is a tuple containing the source (c) file and header (h) file.For example when using our Quick usage grammar, this is the output when runningmy_grammar.export_c():/** grammar.c** This grammar is generated using the Grammar.export_c() method and* should be used with the libcleri module.** Source class: MyGrammar* Created at: 2016-05-09 12:16:49*/#include"grammar.h"#include<stdio.h>#define CLERI_CASE_SENSITIVE 0#define CLERI_CASE_INSENSITIVE 1#define CLERI_FIRST_MATCH 0#define CLERI_MOST_GREEDY 1cleri_grammar_t*compile_grammar(void){cleri_t*r_name=cleri_regex(CLERI_GID_R_NAME,"^(?:\"(?:[^\"]*)\")+");cleri_t*k_hi=cleri_keyword(CLERI_GID_K_HI,"hi",CLERI_CASE_INSENSITIVE);cleri_t*START=cleri_sequence(CLERI_GID_START,2,k_hi,r_name);cleri_grammar_t*grammar=cleri_grammar(START,"^\\w+");returngrammar;}and the header file.../** grammar.h** This grammar is generated using the Grammar.export_c() method and* should be used with the libcleri module.** Source class: MyGrammar* Created at: 2016-05-09 12:16:49*/#ifndef CLERI_EXPORT_GRAMMAR_H_#define CLERI_EXPORT_GRAMMAR_H_#include<grammar.h>#include<cleri/cleri.h>cleri_grammar_t*compile_grammar(void);enumcleri_grammar_ids{CLERI_NONE,// used for objects with no nameCLERI_GID_K_HI,CLERI_GID_R_NAME,CLERI_GID_START,CLERI_END// can be used to get the enum length};#endif/* CLERI_EXPORT_GRAMMAR_H_ */export_gosyntax:Grammar().export_go(go_template=Grammar.GO_TEMPLATE,go_indent='\t',go_package='grammar')Optional keyword arguments:go_template: Template String used for the export. You might want to look at the default string which can be found at Grammar.GO_TEMPLATE.go_indent: indentation used in the Go file. (default: one tab)go_package: Name of the go package. (default: 'grammar')For example when using our Quick usage grammar, this is the output when runningmy_grammar.export_go():packagegrammar// This grammar is generated using the Grammar.export_go() method and// should be used with the goleri module.//// Source class: MyGrammar// Created at: 2017-03-14 19:07:09import("regexp""github.com/cesbit/goleri")// Element indentifiersconst(NoGid=iotaGidKHi=iotaGidRName=iotaGidSTART=iota)// MyGrammar returns a compiled goleri grammar.funcMyGrammar()*goleri.Grammar{rName:=goleri.NewRegex(GidRName,regexp.MustCompile(`^(?:"(?:[^"]*)")+`))kHi:=goleri.NewKeyword(GidKHi,"hi",false)START:=goleri.NewSequence(GidSTART,kHi,rName,)returngoleri.NewGrammar(START,regexp.MustCompile(`^\w+`))}export_javasyntax:Grammar().export_java(java_template=Grammar.JAVA_TEMPLATE,java_indent=' '*4,java_package=None,is_public=True)Optional keyword arguments:java_template: Template String used for the export. You might want to look at the default string which can be found at Grammar.JAVA_TEMPLATE.java_indent: indentation used in the Java file. (default: four spaces)java_package: Name of the Java package or None when no package is specified. (default: None)is_public: Class and constructor are defined as public when True, else they will be defined as package private.For example when using our Quick usage grammar, this is the output when runningmy_grammar.export_java():/*** This grammar is generated using the Grammar.export_java() method and* should be used with the jleri module.** Source class: MyGrammar* Created at: 2018-07-04 12:12:34*/importjleri.Grammar;importjleri.Element;importjleri.Sequence;importjleri.Regex;importjleri.Keyword;publicclassMyGrammarextendsGrammar{enumIds{K_HI,R_NAME,START}privatestaticfinalElementR_NAME=newRegex(Ids.R_NAME,"^(?:\"(?:[^\"]*)\")+");privatestaticfinalElementK_HI=newKeyword(Ids.K_HI,"hi",false);privatestaticfinalElementSTART=newSequence(Ids.START,K_HI,R_NAME);publicMyGrammar(){super(START,"^\\w+");}}export_pysyntax:Grammar().export_py(py_module_name='pyleri',py_template=Grammar.PY_TEMPLATE,py_indent=' '*4)Optional keyword arguments:py_module_name: Name of the Pyleri Module. (default: 'pyleri')py_template: Template String used for the export. You might want to look at the default string which can be found at Grammar.PY_TEMPLATE.py_indent: indentation used in the Python file. (default: 4 spaces)For example when using our Quick usage grammar, this is the output when runningmy_grammar.export_py():"""This grammar is generated using the Grammar.export_py() method andshould be used with the pyleri python module.Source class: MyGrammarCreated at: 2017-03-14 19:14:51"""importrefrompyleriimportSequencefrompyleriimportKeywordfrompyleriimportGrammarfrompyleriimportRegexclassMyGrammar(Grammar):RE_KEYWORDS=re.compile('^\\w+')r_name=Regex('^(?:"(?:[^"]*)")+')k_hi=Keyword('hi')START=Sequence(k_hi,r_name)ResultThe result of theparse()method contains 4 properties that will be explained next. A functionas_str(translate=None)is also available which will
show the result as a string. Thetranslateargument should be a function which accepts an element as argument. This function can be used to
return custom strings for certain elements. If the return value oftranslateisNonethen the function will fall try to generate a string value. If
the return value is an empty string, the value will be ignored.Example of translate functions:# In case a translation function returns an empty string, no text is useddeftranslate(elem):return''# as a result you get something like: 'error at position x'# Text may be returned based on giddeftranslate(elem):ifelemissome_elem:return'A'# something like: error at position x, expecting: Aelifelemisother_elem:return''# other_elem will be ignoredelse:returnNone# normal parsing# A translate function can be used as follow:print(my_grammar.parse('some string').as_str(translate=translate))is_validis_validreturns a boolean value,Truewhen the given string is valid according to the given grammar,Falsewhen not valid.Let us take the example from Quick usage.res=my_grammar.parse('bye "Iris"')print(res.is_valid)# => FalsePositionposreturns the position where the parser had to stop. (whenis_validisTruethis value will be equal to the length of the given string withstr.rstrip()applied)Let us take the example from Quick usage.result=my_grammar.parse('hi Iris')print(res.is_valid,result.pos)# => False, 3Treetreecontains the parse tree. Even whenis_validisFalsethe parse tree is returned but will only contain results as far as parsing has succeeded. The tree is the root node which can include severalchildrennodes. The structure will be further clarified in the following example which explains a way of visualizing the parse tree.Example:importjsonfrompyleriimportChoicefrompyleriimportGrammarfrompyleriimportKeywordfrompyleriimportRegexfrompyleriimportRepeatfrompyleriimportSequence# Create a Grammar Class to define your languageclassMyGrammar(Grammar):r_name=Regex('(?:"(?:[^"]*)")+')k_hi=Keyword('hi')k_bye=Keyword('bye')START=Repeat(Sequence(Choice(k_hi,k_bye),r_name))# Returns properties of a node object as a dictionary:defnode_props(node,children):return{'start':node.start,'end':node.end,'name':node.element.nameifhasattr(node.element,'name')elseNone,'element':node.element.__class__.__name__,'string':node.string,'children':children}# Recursive method to get the children of a node object:defget_children(children):return[node_props(c,get_children(c.children))forcinchildren]# View the parse tree:defview_parse_tree(res):start=res.tree.children[0]\ifres.tree.childrenelseres.treereturnnode_props(start,get_children(start.children))if__name__=='__main__':# Compile your grammar by creating an instance of the Grammar Class:my_grammar=MyGrammar()res=my_grammar.parse('hi "pyleri" bye "pyleri"')# The parse tree is visualized as a JSON object:print(json.dumps(view_parse_tree(res),indent=2))Part of the output is shown below.{"start":0,"end":23,"name":"START","element":"Repeat","string":"hi \"pyleri\" bye \"pyleri\"","children":[{"start":0,"end":11,"name":null,"element":"Sequence","string":"hi \"pyleri\"","children":[{"start":0,"end":2,"name":null,"element":"Choice","string":"hi","children":[{"start":0,"end":2,"name":"k_hi","element":"Keyword","string":"hi","children":[]}]},{"start":3,"end":11,"name":"r_name","element":"Regex","string":"\"pyleri\"","children":[]}"...""..."A node contains 5 properties that will be explained next:startproperty returns the start of the node object.endproperty returns the end of the  node object.elementreturns theElement's type (e.g. Repeat, Sequence, Keyword, etc.). An element can be assigned to a variable; for instance in the example aboveKeyword('hi')was assigned tok_hi. Withelement.namethe assigned namek_hiwill be returned. Note that it is not a given that an element is named; in our exampleSequencewas not assigned, thus in this case the element has no attributename.stringreturns the string that is parsed.childrencan return a node object containing deeper layered nodes provided that there are any. In our example the root node has an element typeRepeat(), starts at 0 and ends at 24, and it has twochildren. These children are node objects that have both an element typeSequence, start at 0 and 12 respectively, and so on.Expectingexpectingreturns a Python set() containing elements which pyleri expects atpos. Even ifis_validis true there might be elements in this set, for example when anOptional()element could be added to the string. "Expecting" is useful if you want to implement things like auto-completion, syntax error handling, auto-syntax-correction etc. The following example will illustrate a way of implementation.Example:importreimportrandomfrompyleriimportChoicefrompyleriimportGrammarfrompyleriimportKeywordfrompyleriimportRepeatfrompyleriimportSequencefrompyleriimportend_of_statement# Create a Grammar Class to define your language.classMyGrammar(Grammar):RE_KEYWORDS=re.compile(r'\S+')r_name=Keyword('"pyleri"')k_hi=Keyword('hi')k_bye=Keyword('bye')START=Repeat(Sequence(Choice(k_hi,k_bye),r_name),mi=2)# Print the expected elements as a indented and numbered list.defprint_expecting(node_expecting,string_expecting):forloop,einenumerate(node_expecting):string_expecting='{}\n\t({}){}'.format(string_expecting,loop,e)print(string_expecting)# Complete a string until it is valid according to the grammar.defauto_correction(string,my_grammar):node=my_grammar.parse(string)print('\nParsed string:{}'.format(node.tree.string))ifnode.is_valid:string_expecting='String is valid.\nExpected: 'print_expecting(node.expecting,string_expecting)else:string_expecting='String is NOT valid.\nExpected: '\ifnotnode.pos\else'String is NOT valid.\nAfter "{}" expected: '.format(node.tree.string[:node.pos])print_expecting(node.expecting,string_expecting)selected=random.choice(list(node.expecting))string='{}{}'.format(node.tree.string[:node.pos],selectedifselectedisnotend_of_statementelse'')auto_correction(string,my_grammar)if__name__=='__main__':# Compile your grammar by creating an instance of the Grammar Class.my_grammar=MyGrammar()string='hello "pyleri"'auto_correction(string,my_grammar)Output:Parsed string: hello "pyleri"
String is NOT valid.
Expected:
        (1) hi
        (2) bye

Parsed string:  bye
String is NOT valid.
After " bye" expected:
        (1) "pyleri"

Parsed string:  bye "pyleri"
String is NOT valid.
After " bye "pyleri"" expected:
        (1) hi
        (2) bye

Parsed string:  bye "pyleri" hi
String is NOT valid.
After " bye "pyleri" hi" expected:
        (1) "pyleri"

Parsed string:  bye "pyleri" hi "pyleri"
String is valid.
Expected:
        (1) hi
        (2) byeIn the above example we parsed an invalid string according to the grammar class. Theauto-correction()method that we built for this example combines all properties from theparse()to create a valid string. The output shows every recursion of theauto-correction()method and prints successively the set of expected elements. It takes one randomly and adds it to the string. When the string corresponds to the grammar, the propertyis_validwill returnTrue. Notably theexpectingproperty still contains elements even if theis_validreturnedTrue. The reason in this example is due to theRepeatelement.ElementsPyleri has several elements which are all subclasses ofElementand can be used to create a grammar.Keywordsyntax:Keyword(keyword,ign_case=False)The parser needs to match the keyword which is just a string. When matching keywords we need to tell the parser what characters are allowed in keywords. By default Pyleri uses^\w+which is both in Python and JavaScript equal to^[A-Za-z0-9_]+. We can overwrite the default by settingRE_KEYWORDSin the grammar. Keyword() accepts one keyword argumentign_caseto tell the parser if we should match case insensitive.Example:classTicTacToe(Grammar):# Let's allow keywords with alphabetic characters and dashes.RE_KEYWORDS=re.compile('^[A-Za-z-]+')START=Keyword('tic-tac-toe',ign_case=True)ttt_grammar=TicTacToe()ttt_grammar.parse('Tic-Tac-Toe').is_valid# => TrueRegexsyntax:Regex(pattern,flags=0)The parser compiles a regular expression using theremodule. The current version of pyleri has only support for there.IGNORECASEflag.
See theQuick usageexample for how to useRegex.Tokensyntax:Token(token)A token can be one or more characters and is usually used to match operators like+,-,//and so on. When we parse a string object where pyleri expects an element, it will automatically be converted to aToken()object.Example:classNi(Grammar):t_dash=Token('-')# We could just write delimiter='-' because# any string will be converted to Token()START=List(Keyword('ni'),delimiter=t_dash)ni=Ni()ni.parse('ni-ni-ni-ni-ni').is_valid# => TrueTokenssyntax:Tokens(tokens)Can be used to register multiple tokens at once. Thetokensargument should be a string with tokens separated by spaces. If given tokens are different in size the parser will try to match the longest tokens first.Example:classNi(Grammar):tks=Tokens('+ - !=')START=List(Keyword('ni'),delimiter=tks)ni=Ni()ni.parse('ni + ni != ni - ni').is_valid# => TrueSequencesyntax:Sequence(element,element,...)The parser needs to match each element in a sequence.Example:classTicTacToe(Grammar):START=Sequence(Keyword('Tic'),Keyword('Tac'),Keyword('Toe'))ttt_grammar=TicTacToe()ttt_grammar.parse('Tic Tac Toe').is_valid# => TrueChoicesyntax:Choice(element,element,...,most_greedy=True)The parser needs to choose between one of the given elements. Choice accepts one keyword argumentmost_greedywhich isTrueby default. Whenmost_greedyis set toFalsethe parser will stop at the first match. WhenTruethe parser will try each element and returns the longest match. Settingmost_greedytoFalsecan provide some extra performance. Note that the parser will try to match each element in the exact same order they are parsed to Choice.Example: let us useChoiceto modify the Quick usage example to allow the string 'bye "Iris"'classMyGrammar(Grammar):r_name=Regex('(?:"(?:[^"]*)")+')k_hi=Keyword('hi')k_bye=Keyword('bye')START=Sequence(Choice(k_hi,k_bye),r_name)my_grammar=MyGrammar()my_grammar.parse('hi "Iris"').is_valid# => Truemy_grammar.parse('bye "Iris"').is_valid# => TrueRepeatsyntax:Repeat(element,mi=0,ma=None)The parser needs at leastmielements and at mostmaelements. Whenmais set toNonewe allow unlimited number of elements.mican be any integer value equal or higher than 0 but not larger thenma.Example:classNi(Grammar):START=Repeat(Keyword('ni'))ni=Ni()ni.parse('ni ni ni ni ni').is_valid# => TrueIt is not allowed to bind a name to the same element twice and Repeat(element, 1, 1) is a common solution to bind the element a second (or more) time(s).For example consider the following:classMyGrammar(Grammar):r_name=Regex('(?:"(?:[^"]*)")+')# Raises a SyntaxError because we try to bind a second time.r_address=r_name# WRONG# Instead use Repeatr_address=Repeat(r_name,1,1)# RIGHTListsyntax:List(element,delimiter=',',mi=0,ma=None,opt=False)List is like Repeat but with a delimiter. A comma is used as default delimiter but any element is allowed. When a string is used as delimiter it will be converted to aTokenelement.miandmawork exactly like with Repeat. An optional keyword argumentoptcan be set toTrueto allow the list to end with a delimiter. By default this is set toFalsewhich means the list has to end with an element.Example:classNi(Grammar):START=List(Keyword('ni'))ni=Ni()ni.parse('ni, ni, ni, ni, ni').is_valid# => TrueOptionalsyntax:Optional(element)The parser looks for an optional element. It is like usingRepeat(element, 0, 1)but we encourage to useOptionalsince it is more readable. (and slightly faster)Example:classMyGrammar(Grammar):r_name=Regex('(?:"(?:[^"]*)")+')k_hi=Keyword('hi')START=Sequence(k_hi,Optional(r_name))my_grammar=MyGrammar()my_grammar.parse('hi "Iris"').is_valid# => Truemy_grammar.parse('hi').is_valid# => TrueRefsyntax:Ref()The grammar can make a forward reference to make recursion possible. In the example below we create a forward reference to START but note that
a reference to any element can be made.Warning: A reference is not protected against testing the same position in
a string. This could potentially lead to an infinite loop.
For example:r=Ref()r=Optional(r)# DON'T DO THISUsePrioif such recursive construction is required.Example:classNestedNi(Grammar):START=Ref()ni_item=Choice(Keyword('ni'),START)START=Sequence('[',List(ni_item),']')nested_ni=NestedNi()nested_ni.parse('[ni, ni, [ni, [], [ni, ni]]]').is_valid# => TruePriosyntax:Prio(element,element,...)Choose the first match from the prio elements and allowTHISfor recursive operations. WithTHISwe point to thePrioelement. Probably the example below explains howPrioandTHIScan be used.Note: Use aRefwhen possible.
APrioelement is required when the same position in a string is potentially
checked more than once.Example:classNi(Grammar):k_ni=Keyword('ni')START=Prio(k_ni,# '(' and ')' are automatically converted to Token('(') and Token(')')Sequence('(',THIS,')'),Sequence(THIS,Keyword('or'),THIS),Sequence(THIS,Keyword('and'),THIS))ni=Ni()ni.parse('(ni or ni) and (ni or ni)').is_valid# => True