Package: tokenstream
tokenstreamA versatile token stream for handwritten parsers.fromtokenstreamimportTokenStreamdefparse_sexp(stream:TokenStream):"""A basic S-expression parser."""withstream.syntax(brace=r"\(|\)",number=r"\d+",name=r"\w+"):brace,number,name=stream.expect(("brace","("),"number","name")ifbrace:return[parse_sexp(stream)for_instream.peek_until(("brace",")"))]elifnumber:returnint(number.value)elifname:returnname.valueprint(parse_sexp(TokenStream("(hello (world 42))")))# ['hello', ['world', 42]]IntroductionWriting recursive-descent parsers by hand can be quite elegant but it's often a bit more verbose than expected, especially when it comes to handling indentation and reporting proper syntax errors. This package provides a powerful general-purpose token stream that addresses these issues and more.FeaturesDefine the set of recognizable tokens dynamically with regular expressionsTransparently skip over irrelevant tokensExpressive API for matching, collecting, peeking, and expecting tokensClean error reporting with line numbers and column numbersContextual support for indentation-based syntaxCheckpoints for backtracking parsersWorks well with Python 3.10+ match statementsCheck out theexamplesdirectory for practical examples.InstallationThe package can be installed withpip.pipinstalltokenstreamGetting startedYou can define tokens with thesyntax()method. The keyword arguments associate regular expression patterns to token types. The method returns a context manager during which the specified tokens will be recognized.stream=TokenStream("hello world")withstream.syntax(word=r"\w+"):print([token.valuefortokeninstream])# ['hello', 'world']Check out the fullAPI referencefor more details.Expecting tokensThe token stream is iterable and will yield all the extracted tokens one after the other. You can also retrieve tokens from the token stream one at a time by using theexpect()method.stream=TokenStream("hello world")withstream.syntax(word=r"\w+"):print(stream.expect().value)# "hello"print(stream.expect().value)# "world"Theexpect()method lets you ensure that the extracted token matches a specified type and will raise an exception otherwise.stream=TokenStream("hello world")withstream.syntax(number=r"\d+",word=r"\w+"):print(stream.expect("word").value)# "hello"print(stream.expect("number").value)# UnexpectedToken: Expected number but got word 'world'Filtering the streamNewlines and whitespace are ignored by default. You can reject interspersed whitespace by intercepting the built-innewlineandwhitespacetokens.stream=TokenStream("hello world")withstream.syntax(word=r"\w+"),stream.intercept("newline","whitespace"):print(stream.expect("word").value)# "hello"print(stream.expect("word").value)# UnexpectedToken: Expected word but got whitespace ' 'The opposite of theintercept()method isignore(). It allows you to ignore tokens and handle comments pretty easily.stream=TokenStream("""# this is a commenthello # also a commentworld""")withstream.syntax(word=r"\w+",comment=r"#.+$"),stream.ignore("comment"):print([token.valuefortokeninstream])# ['hello', 'world']IndentationTo enable indentation you can use theindent()method. The stream will now yield balanced pairs ofindentanddedenttokens when the indentation changes.source="""helloworld"""stream=TokenStream(source)withstream.syntax(word=r"\w+"),stream.indent():stream.expect("word")stream.expect("indent")stream.expect("word")stream.expect("dedent")To prevent some tokens from triggering unwanted indentation changes you can use theskipargument.source="""hello# some commentworld"""stream=TokenStream(source)withstream.syntax(word=r"\w+",comment=r"#.+$"),stream.indent(skip=["comment"]):stream.expect("word")stream.expect("comment")stream.expect("indent")stream.expect("word")stream.expect("dedent")CheckpointsThecheckpoint()method returns a context manager that resets the stream to the current token at the end of thewithstatement. You can use the returnedcommit()function to keep the state of the stream at the end of thewithstatement.stream=TokenStream("hello world")withstream.syntax(word=r"\w+"):withstream.checkpoint():print([token.valuefortokeninstream])# ['hello', 'world']withstream.checkpoint()ascommit:print([token.valuefortokeninstream])# ['hello', 'world']commit()print([token.valuefortokeninstream])# []Match statementsMatch statements make it very intuitive to process tokens extracted from the token stream. If you're using Python 3.10+ give it a try and see if you like it.fromtokenstreamimportTokenStream,Tokendefparse_sexp(stream:TokenStream):"""A basic S-expression parser that uses Python 3.10+ match statements."""withstream.syntax(brace=r"\(|\)",number=r"\d+",name=r"\w+"):matchstream.expect_any(("brace","("),"number","name"):caseToken(type="brace"):return[parse_sexp(stream)for_instream.peek_until(("brace",")"))]caseToken(type="number")asnumber:returnint(number.value)caseToken(type="name")asname:returnname.valueContributingContributions are welcome. Make sure to first open an issue discussing the problem or the new feature before creating a pull request. The project usespoetry.$poetryinstallYou can run the tests withpoetry run pytest.$poetryrunpytestThe project must type-check withpyright. If you're using VSCode thepylanceextension should report diagnostics automatically. You can also install the type-checker locally withnpm installand run it from the command-line.$npmrunwatch
$npmruncheck
$npmrunverifytypesThe code follows theblackcode style. Import statements are sorted withisort.$poetryrunisorttokenstreamexamplestests
$poetryrunblacktokenstreamexamplestests
$poetryrunblack--checktokenstreamexamplestestsLicense -MIT