Package: sdataflow
a small and simple language within the projectsblog.Installpip install sdataflowConceptssdataflowprovides:A small and simple language to define the relation of entities. Anentityis a logic unit defined by user(i.e. a data processing
function), it generates some kind ofoutcomeas a respond to some
kind of inputoutcome(which might be generated by other
Entity). Relations of entities forms adataflow.An command line programsdataflowgenerates html file for
debugging.A scheduler automatically runs entities and ships outcome to its
destination.LanguageTutorialLet’s start with a simplest case(one-to-onerelation):A --> Bwhere entityBaccepts outcome ofAas its input.To define aone-to-moreormore-to-onerelation:# one-to-more
A --> B
A --> C
A --> D

# more-to-one
B --> A
C --> A
D --> Awhere in theone-to-morecase, copies of outcome ofAcould be
passed toB,CandD. In themore-to-onecase, outcomes
ofB,CandDwould be passed toA.And here’s the form ofoutcome dispatching, that is, a mechanism of
sending different kinds of outcome of an entity to different
destinations. For instance, entityAgenerates two kinds of outcome,
say[type1]and[type2], and pass outcomes of[type1]toB, outcomes of[type2]toC:# one way.
A --> [type1]
A --> [type2]
[type1] --> B
[type2] --> C

# another way.
A --[type1]--> B
A --[type2]--> Cwhere identifier embraced in brackets(i.e.[type1]) represents the
name of outcome. In contrast to the form of outcome dispatching,A-->Bwould simple pass outcome ofA, with default nameA(the name of entity generates the outcome), to entityB.
Essentially, above form(statement contains brackets) overrides the name
of outcome, and acts like a filter for outcome dispatching.Outcome could be used to defineone-to-more,more-to-onerelations as well, in the same way discussed above:# one-to-more example.
A --> [type1]
A --> [type2]
[type1] --> B
[type1] --> C
[type2] --> D
[type2] --> E

# more-to-one example.
A --> [type1]
B --> [type1]
[type1] --> CAfter loading all user defined dataflow, there are basically two steps
of analysis will be applied:Build a DAG of dataflow. Break if error happens(i.e. syntax error,
cyclic path).Apply topology sort to DAG to get the linear ordering of entity
invocation.Lexical RulesARROW          : re.escape('-->')
DOUBLE_HYPHENS : re.escape('--')
BRACKET_LEFT   : re.escape('[')
BRACKET_RIGHT  : re.escape(']')
ID             : r'\w+'The effect of above rules would be equivalent as if passing such rules
to Python’sremodule with the flagUNICODEbeing set.CFGsstart : stats

stats : stats single_stat
      | empty

single_stat : entity_to_entity
            | entity_to_outcome
            | outcome_to_entity

entity_to_entity : ID general_arrow ID

general_arrow : ARROW
              | DOUBLE_HYPHENS outcome ARROW

outcome : BRACKET_LEFT ID BRACKET_RIGHT

entity_to_outcome : ID ARROW outcome

outcome_to_entity : outcome ARROW IDCommand-line programAfter installsdataflowthroughpip, user can invoke a
command-line programsdataflow. Synopsis ofsdataflowis simple:Usage:
    sdataflow <file>User could pass the file path of datafow definition tosdataflow,
then the program will parse the file, analyse the dataflow and finally
generate a html file. Ues a browser to open such html file(based on
projectmermaid), and then, you
get a graphic representation of your dataflow!An example is given for illustration:$ cat example.sd
A --[odd]--> B
A --[even]--> C
B --> D
C --> D
$ sdataflow example.sd
$ ls
example.html example.sdUes a browser to openexample.html:screen shot 2015-04-28 at 12 02 58 amAPIForm of CallbackAs mentioned above, an entity stands for a user defined logic unit.
Hence, after defining the relations of entities in the language
discussed above, user should defines a set of callbacks, corresponding
to each entity in the definition.A callback is acallable(function, generator, bound method) that
returnsNone(i.e. a function with noreturnstatement), or an
iterable object of which the element is a (key, value) tuple, with key
as the name of outcome and value as user defined object. Argument list
of such callable could be:An empty list, meaning that such callback accept no data.An one-element list.Code fragment for illustration:# normal function returns `None`, with empty argument list.deffunc1():pass# normal function return `None`, with one-element argument list.deffunc2(items):forname_of_outcome,objinitems:# do something.# normal function return elements, with one-element argument list.deffunc3(items):# ignore `items`data=[('some outcome name',i)foriinrange(10)]returndata# generator yield element, with one-element argument list.defgen1(items):# ignore `items`foriinrange(10):yield'some outcome name',iclassExampleClass(object):@classmethoddefmethod1(cls):pass@classmethoddefmethod2(cls,items):passdefmethod3(self):passdefmethod4(self,items):pass# class bound method, with empty argument list.ExampleClass.method1# class bound method, with one-element argument list.ExampleClass.method2example_instance=ExampleClass()# class bound method, with empty argument list.example_instance.method3# class bound method, with one-element argument list.example_instance.method4Note that the name of outcome is the string embraced in
brackets(notincluding the brackets).All In One Interfacesdataflowprovides a classsdataflow.DataflowHandlerto parsedoc(a string represents the relations of entities), register
callbacks and schedule the execution of callbacks.class DataflowHandler
    __init__(self, doc, name_callback_mapping=None)
        `doc`: unicode or utf-8 encoded binary data.
        `name_callback_mapping`: a dict of (`name`, `callback`) pairs. `name`
        could be unicode or utf-8 encoded binary data. `callback` is a function
        or generator. `name_callback_mapping` could be `None`, since callback
        can be registered by function decorator(see next section).

    run(self)
        Automatically execute all registered callbacks.Example:fromsdataflowimportDataflowHandlerfromsdataflow.callbackimportcreate_data_wrapperdoc=('A --[odd]--> B ''A --[even]--> C ''B --> D ''C --> D ')defa():odd=create_data_wrapper('odd')even=create_data_wrapper('even')foriinrange(1,10):ifi%2==0:yieldeven(i)else:yieldodd(i)defb(items):default=create_data_wrapper('B')# remove 1.foroutcome_name,numberinitems:ifnumber==1:continueyielddefault(number)defc(items):default=create_data_wrapper('C')# remove 2.foroutcome_name,numberinitems:ifnumber==2:continueyielddefault(number)defd(items):numbers={ifor_,iinitems}assertset(range(3,10))==numbersname_callback_mapping={'A':a,'B':b,'C':c,'D':d,}# parse `doc`, register `a`, `b`, `c`, `d`.handler=DataflowHandler(doc,name_callback_mapping)# execute callbacks.handler.run()In above example,Agenerates numbers in the range of 1 to 9, of
which the odd numbers(1, 3, 5, 7, 9) are sent toB, the even
numbers(2, 4, 6, 8) are sent toC. ThenBremoves number 1 and
sends the rest(3, 5, 7, 9) toD, whileCremoves number 2 and
sends the rest(4, 6, 8) toD. Finally,Dreceives outcomes of
bothCandD, and make sure that is equal toset(range(3, 10)).Use Decorator To Register Normal Functionsdataflow.callback.register_callbackis a function decorator with
signature:register_callback(entity_name, *outcome_names)whereentity_namecould be an unicode or utf-8 encoded binary
string, indicating the entity to which the function should be
registered. Ifoutcome_namesis given, the decorator would inject
severalsdataflow.callback.create_data_wrappergenerated data
wrapper to the function being decorated.Example:@register_callback('A')defzero_arg():return0@register_callback('C')defone_arg(items):return1DataflowHandler(doc)wherezero_argis registered to entityA,one_argis
registered to entityB. Note that as mentioned above, second
parameter ofDataflowHandlercan be ignored.When names of decorator registered callback conflict with names ofname_callback_mapping, the second parameter ofDataflowHandler,
callbacks inname_callback_mappingwill be accepted, and callbacks
registered by function decorator will be discarded. For example:@register_callback('A')defzero_arg():return0@register_callback('C')defshould_not_be_registered(items):return1defone_arg(items):return42DataflowHandler(doc,{'C':one_arg})whereone_argwill be registered instead ofshould_not_be_registered.Example of function injection:@register_callback('A','type1','type2')deffunc():returnfunc.type1(1),func.type2(2)assert(('type1',1),('type2',2),)==func()Be careful to applyregister_callbackto things other thanfunction, let’s say, you want to register a class method:classExample(object):# wrong, `classmethod` is not bound.@register_callback('A')@classmethoddeffunc(cls):pass# try following code instead.register_callback('A')(Example.func)Pure Interface ofsdataflowLanguagesdataflow.lang.parsecan be used to parse the definition of
dataflow:parse(doc)
    input: `doc` with type of six.binary_type or six.text_type.
    output: linear ordering and root nodes of dataflow.parsereturns a 2-tuple, with the first element is a list of linear
ordering of dataflow, and the second element is a list of root nodes of
the forest.