Package: wordstreamer
wordstreamer is a tiny generic text-generation module. It is a set of convenient generator-based tools to generate any text.Be it code, markup or anything else, wordstreamer got you covered.Streams / generators are perfect for text generations.Those are easy to use, easily combined, and have a great speed and memory footprint in concurrent / stream-based workflows (such as webservers).Features:Build custom renderablesAn operation priority system with optional custom wrapping logicRender as string / bytes, as string / byte streams or as text / binary filesInsert arbitrary markers to pass information upstreamUse context to tweak rendering as you needSeveral common generic renderables included in startkitBuild custom stream processorsProcess streams of tokens in any waySeveral common processors included in stream_utilsHave fun with strings attachedInstallationJust installwordstreamermodule from PyPIExamplesCheck outexamplesfolder for examplesMath expressionsAlmost correct json.dumps implementationDocsCoreCore module (wordstreamer.core) consists of several user-facing entities:Renderablebase class,MarkerandRendererclasses andget_renderfunction.Other classes (ContextandStreamFile) are internal, but fairly easy to use if needed.Your basic flow is:Create severableRenderablefor your needsCreate an instance ofRendererwith context payloadUse one ofRenderer's methods to output the renderable in preferrable formatRenderablewordstreamer.core.Renderableis the main class of the module.Your basic custom renderable has to implementstreammethod that would return a stream of tokens.Here's a simple renderable that renders a list-like string from a range, simulatingstr(list(range(...)))fromtypingimportIterablefromwordstreamerimportRenderable,Context,TokenStreamfromwordstreamer.stream_utilsimportomit_endclassRange(Renderable):def__init__(self,value:range):self.value=value# let's make a separate method for conveniencedefnumbers(self,context:Context)->TokenStream:forninself.value:yieldstr(n)yield","yield" "defstream(self,context:Context)->TokenStream:yield"["yield fromomit_end(self.numbers(context))yield"]"(Note: we could have doneyield ", ", but that would make stream manipulation less effective (e.g. stripping unwanted spaces))Then, to render it, userender_string:# prints "[2, 4, 6, 8]"print(Range(range(2,9,2)).render_string())Functional renderablesSometimes, you need simple renderables defining only.stream()method, similar to React functional components.Something like this:defrender_sum(left:float,right:float)->TokenStream:yieldstr(left)yield" + "yieldstr(right)wordstreamer defines a convenience wrapper for that,make_renderable.Just use it as a decorator and addContextas a first positional argument:fromwordstreamerimportmake_renderable,Context@make_renderabledefrender_sum(context:Context,left:float,right:float)->TokenStream:yieldstr(left)yield" + "yieldstr(right)And use it like any other renderable, omittingcontextargument:print(render_sum(4,5).render_string())The downside of that approach is that you get a basic plain Renderable, which makes it impossible to build a complex type hierarchyRenderer and ContextWhile usingrenderable.render_string()is totally fine, you can easily dive into the rendering flow.You have noticed this mysterious Context value going inside function arguments. This is a payload that allows you to pass data from top to bottom, and vice versa.It can be passed intorenderable.render_string(context: Payload), wherePayloadis a dictionary withstrkeys and arbitrary values.Inside.streamfunctions, this payload becomes an instance ofContext, which is a simple wrapper that supports attribute access:classMyRenderable(Renderable):defstream(self,context:Context)->TokenStream:ifcontext.left:yieldstr(context.left)ifcontext.op:yieldstr(context.op)ifcontext.right:yieldstr(context.right)MyRenderable().render_string({"left":4,"op":"+","right":5})# "4+5"Context is also mutable andderivable. It means you can derive a new context from an old one and a new payload, passing it to bottom components:...defstream(self,context:Context)->TokenStream:level=context.levelor0subcontext=context.derive(level=level+1)yield"-"*levelyield fromself.child.render(subcontext)...When you need a more granular control over rendering steps and output, useRendererclass. Let's start withRenderable.render_strsource code:defrender_string(self,context:Payload|None=None):"""Render component with a provided context. Check `Renderer` class for advanced rendering"""returnRenderer(context).render_string(self)Oh no, we were fooled! Anyway, theRendererclass takes care of transforming the token stream into some other form, be it a string, bytestring, or even a file.To use it, first create a renderer:fromwordstreamerimportRendererrenderer=Renderer({"some_key":1,"another_key":"secret_passwordQWERTY"})Then, you can produce the output in 7 formats:As aStream[Token]withrenderer.stream(payload)As aStream[str]withrenderer.str_stream(payload)(takes a TokenStream and drops allMarkerinstances)As aStream[bytes]withrenderer.byte_stream(payload)(takes a Stream[str] and encodes in as UTF-8)Asbyteswithrenderer.render_bytes(payload)(joins aStream[bytes])Asstrwithrenderer.render_string(payload)(joins aStream[str])As a text file withrenderer.as_file(payload)(makes a file wrapper aroundStream[str])As a binary file withrenderer.as_binary_file(payload)(makes a file wrapper aroundStream[bytes])Combining renderablesAny complex text builder has nesting.Due to stream-based flow, one can easily combine renderables.Let's make a simple math expression renderer. We will support numbers, five binary operations (+,-,*,/, and**), and unary-.First, let's make a base class to:define common methodsbuild a type hierarchy (this is optional, but it's extremely convenient and straightforward when you have a complex syntax).from__future__importannotationsfromwordstreamerimportRenderable# we will need this laterpriorities:dict[str,int]={"+":0,"-":0,"*":1,"/":1,"u-":2,"**":3,}classExpression(Renderable):def__add__(self,other:Expression)->BinaryExpression:returnBinaryExpression(self,"+",other)def__sub__(self,other:Expression)->BinaryExpression:returnBinaryExpression(self,"-",other)def__mul__(self,other:Expression)->BinaryExpression:returnBinaryExpression(self,"*",other)def__truediv__(self,other:Expression)->BinaryExpression:returnBinaryExpression(self,"/",other)def__pow__(self,other:Expression)->BinaryExpression:returnBinaryExpression(self,"**",other)def__neg__(self)->BinaryExpression:returnUnaryMinus(self)Then, let's make a renderable for numbers:classNumber(Expression):priority=100def__init__(self,value:float):self.value=valuedefstream(self,context:Context)->TokenStream:yieldstr(self.value)And for composite expressions:classBinaryExpression(Expression):associativity="left"def__init__(self,lhs:Expression,op:str,rhs:Expression):ifop=="**":self.associativity="right"self.priority=priorities[op]self.lhs=lhs.respect_priority(self,side="left")self.rhs=rhs.respect_priority(self,side="right")self.op=opdefstream(self,context:Context)->TokenStream:yield fromself.lhs.stream(context)yield" "yieldself.opyield" "yield fromself.rhs.stream(context)classUnaryMinus(Expression):priority=priorities["u-"]def__init__(self,rhs:Expression):self.rhs=rhs.respect_priority(self)# this is an optional tweak to omit parens if the negative expression is on the right side, as in `4 ** -5` but `(-5) ** 4`defpriority_comparator(self,operation:Renderable,side:str="none")->bool:ifside=="right":returnFalsereturnsuper().priority_comparator(operation,side)defstream(self,context:Context)->TokenStream:yield"-"yield fromself.rhs.stream(context)Finally, we should treat negative numbers the same as a number with an unary minus:defnumber(value:float)->Number|UnaryMinus:ifvalue<0:return-Number(abs(value))returnNumber(value)That's it! Now let's test our simple language:expression=number(6)+number(10)*(number(4.4)**number(-5))**number(4)# 6 + 10 * (4.4 ** -5) ** 4print(expression.render_string())You can find the full code inexamplesPriority magicYou may be wondering, where did()come from in this example? We didn't render them, right?Those parens actually come from therespect_priorityflow.We callexpression.respect_priority(operation, comparator, side)(latter two args are optional), whereexpressionis some nested expression, andoperationis a parent expression.In6 + 5, the whole expression is operation, while6and5are sub-expressions.In this case, there is no need for wrapping (so that(6) + (5)is redundant). But to know that, we have to knowpriority rulesof the language.The default behaviour is simple: having two attributes,priority(int), andassociativity("left" | "right" | "both").If the sub-expression priority is less than the expression priority, we need wrapping:e=number(5)+number(6)e2=e*number(10)print(e2.render_string())# (5 + 6) * 10Here, if we don't wrap, we will get "5 + 6 _ 10", which is not the original intention. But we've defined that+priority is0and_priority is1by assigning it inBinaryExpressionconstructor.So, when we writes * number(10), we doBinaryExpression(5 + 6, *, 10). Let's call this expressionselfIt calls(5 + 6).respect_priority(self, side="left"), and, because(5 + 6)priority is lower than that ofself, it wraps.If the priority is the same, default comparator checks the associativity. If the side is "none" or associativity is "both", it doesn't wrap.Then, if side is not the same as associativity, it does wrap, like with**:e=number(5)+number(6)e2=number(10)-eprint(e2.render_string())# 10 - (5 + 6)Despite operations having the same priority, we need to wrap the expression on the right, because otherwise it will become(10 - 5) + 6(11), instead of10 - (5 + 6)(-1)So, the default behaviour is to adhere to usual binary expression rules, and wrapping with built-in renderable fromstart kit, Parens.You can easily customize how wrapping is done by redefiningRenderable.wrap(self) -> Renderableon your renderable class, without changing the priority behaviour.The comparator ((self, operation, side = "none") -> bool) can be passed into.respect_priority, or you can redefine.priority_comparatormethodMarkersSometimes you may need to pass some data upwards, and for that TokenStream can include instances of Marker, simple data container:...defstream(self,context:Context)->TokenStream:yield"[",yieldMarker("block-content",data={"values":[1,2,3]})yield"]"...Then, you can watch for markers upstream, in some parent component:fromwordstreamer.utilsimportis_marker...defstream(self,context:Context)->TokenStream:yield"x"yield" "yield"="fortokeninself.block.stream(context):ifis_marker(token)andtoken.key=="block-content":...else:yieldtoken...You can obviously do that with context, but keep in mind that derived context don't pass changes to their parent contextsStart Kit and Stream Utilswordstreamer includes useful building tools in form of two modules:wordstreamer.startkitandwordstreamer.stream_utils.The first is a set of common renderables, while the second is a set of functions to manipulate the stream.Start KitStringifyA simple renderable, taking any object, and rendering it usingstr():fromwordstreamer.startkitimportStringifyStringify(6).render_string()# "6"This is useful when injecting pre-rendered content, for fast prototyping, etc.ParensThis is a generic element that wraps your renderable given two strings, one at the start and one at the end.fromwordstreamer.startkitimportParens,Stringifycontent=Stringify("content")Parens(content).render_string()# "(content)"Parens(content,"[","]").render_string()# "[content]"Parens(content,'"','"').render_string()# '"content"'SeparatedRenders several renderables, injecting a stream of tokens between every renderable:fromwordstreamer.startkitimportSeparated,Stringifynumbers=[Stringify(x)forxinrange(10)]Separated(*numbers,separator=[","," "]).render_string()# "0, 1, 2, 3, 4, 5, 6, 7, 8, 9"If you want to append separator after the stream, passtrail=Truein the constructorBlockRenders a full-blown block, with full customization.By default renders a 4-space indented C-style block (with{}):fromwordstreamer.startkitimportBlock,Stringify,Separatedfromwordstreamerimport(Renderable,TokenStream,make_renderable,Context)@make_renderabledefcounter(context:Context,n:int):foriinrange(n):yieldstr(i)ifi+1<n:yield"\n"head=Stringify("if 10 == 89")body=counter(5)"""if 10 == 89 {01234}"""print(Block(head=head,body=body,).render_string())But, if you want Python-style, let's do that:defpythonic_wrapper(stream:TokenStream)->TokenStream:yield":"yield"\n"yield fromstreamyield"\n""""if 10 == 89:01234"""print(Block(head=head,body=body,wrapper=pythonic_wrapper).render_string())You can omitheadargument altogether, or provide a custom indenter:defstaircase_indenter(stream:TokenStream)->TokenStream:tab_level=1fortokeninstream:yieldtokeniftoken=="\n":yieldtab_level*4*" "tab_level+=1"""if 10 == 89 {01234}"""print(Block(head=head,body=body,indenter=staircase_indenter).render_string())Stream utilswordstreamer.stream_utilscontain a set of 'stream transformers' — functions that manipulate stream (or several) building up a new one.This section is a WIP, but here's what the stream utils module defines:defadd_tab(stream:Stream[Token],tab_char:str="    ",newlines:set[Token]|None=None)->Stream[Token]:"""Adds `tab_char` after any token that is equal to any of the newline tokens in `newlines`.By default, `tab_char` is "    " (4 ASCII spaces U+0020) and `newlines` is {"\\n"}"""defseparated(*streams:Stream[Piece],separator:Stream[Piece],trail:bool=False)->Stream[Piece]:"""Interjects `separator` tokens between streams, outputting a new chained stream.`trail` defines if separator should be added after last token, i.e. if a trailing comma should be inserted"""defprepend(stream:Stream[Piece],*pieces:Piece)->Stream[Piece]:"""Injects tokens before stream"""defappend(stream:Stream[Piece],*pieces:Piece)->Stream[Piece]:"""Injects tokens after stream"""defomit_start(stream:Stream[Piece],count:int=1)->Stream[Piece]:"""Omits `count` tokens from the start of the stream"""defomit_end(stream:Stream[Piece],count:int=1)->Stream[Piece]:"""Omits `count` tokens from the end of the stream"""defconcat(*streams:Stream[Piece]|None)->Stream[Piece]:"""Concatenates several streams"""defwrap(stream:Stream[Piece],prefix:Stream[Piece],postfix:Stream[Piece],)->Stream[Piece]:"""Prepends `prefix` before stream and appends `postfix` after stream"""defstream_noop(stream:Stream[Piece])->Stream[Piece]:"""Leaves stream intact, useful to pass as a no-op transformer"""