Package: basil-parser
BasilSimple Parser Library. Describe the syntax in a JSON file.Features:Easy to useSyntax description in JSON, no complicated notationTransform parse tree to any type you wantHelpful messages when parser runs into errorsLimitations:Does not detect conflictsNot designed for speedHow to install# If you use pippipinstallbasil-parser==0.4.3# If you use PDMpdminstallbasil-parser==0.4.3How to useIf you're unfamiliar with parsers, please read the section onlingo usedbelow.1. Create the syntax JSON for your language.Seethis example, which describes the syntax of a subset of JSON.The JSON should be an object with the following values:akeyword_tokensobject, where the name is the token type insnake_caseand the value is a regex for tokens of this type.aregular_tokensobject, similar to keyword tokens, except names can't overlap. Keyword tokens are considered higher priority.afiltered_tokensarray of strings. Lists token types that should be ignored. This is sometimes useful for getting rid of whitespace.anodesobject, where the name is the node type inTITLE_CASEand the value is an expression describing which configuration of child (tokens and nodes) is allowed, using this syntax:expr | expr: choose one of the expressions(expr expr): combine expressions into oneexpr*: repeat expression 0 or more timesexpr+: repeat expression 1 or more timesexpr?: expression is optionalaroot_nodestring value, indicating the node type of the root of the parse tree.2. Parse and transformIn your python code load the JSON with theFileParser. You only need to load theFileParseronce, regardless of how many files are parsed, because it doesn't change any state when parsing files.The output of the parser is hard to work with, so we transform it into something more workable. To allow the user to transform nodes and tokens into any type, a transformer function must be implemented with signatures as in the below example.For a concrete working example, see howparse_text_and_transform()is used inbasil_json_loads()inthis example.frompathlibimportPathfrombasil.file_parserimportFileParserparser=FileParser(Path("path/to/your/syntax.json"))transformed=parser.parse_text_and_transform("code in your language",node_type="your root_node",node_transformer=node_transformer,token_transformer=token_transformer,)TRANSFORMED_TYPE=...# Some union of typesdefnode_transformer(node_type:str,children:List[TRANSFORMED_TYPE|Token])->TRANSFORMED_TYPE:# Translate to a TRANSFORMED_TYPE value based on the node type and it's children...deftoken_transformer(token:Token)->TRANSFORMED_TYPE|Token:# Translate to TRANSFORMED_TYPE based on token.type or token.value# It is also possible to return `token` as is....3. TestAs usual, testing is optional but recommended. Regexes are tricky. The parser may not match things they way you expect. It is a good idea to test at least all nodes with some inputs that should match and some that should not. Seethis fileas an example.Lingo usedTokenizerChops a file into a list of tokens, which are effectively small chunks of text. When a file is tokenized successfully, every character in the file is part of exactly one token. When the values of all tokens are pasted together, we get the original file.ParserTranslates a list of tokens into a parse-tree consisting of tokens and nodes.TokenA token is a piece of text (value) that has a certain token type. This is usually a single word or a sequence of characters indicating the start, end (such as brackets) or separation (such as comma's) of other parts of the file. Tokens are described by a regex in the syntax JSON.NodeContains a list of children Used to group tokens into a block of code, or data. The types of nodes and tokens that are accepted for a node type are described in the syntax JSON.NameThis project was called "Basil" because many people call their parser library parsley.LicenseThis software is released under theMIT license.