Package: cmdlet
What is cmdlet?Cmdlet provides pipe-like mechanism to cascade functions and generators. It
uses symbol(|) to convert function to Pipe object and cascade them. This
sequence of commands can be executed and evaluated later. Just like pipe
mechanism in Unix shell. For example:fromcmdlet.cmdsimport*# Create piped commands.cmds=range(10)|pipe.filter(lambdax:x>5)|fmt('item#%d')# Execute commands and return the last processed data.run(cmds)# >>> 'item#9'# Execute commands and return processed data in a list.result(cmds)# >>> ['item#6', 'item#7', 'item#8', 'item#9']# Execute commands and return iterator for processed data.fordataincmds:printdata# >>> item#6# >>> item#7# >>> item#8# >>> item#9First, we created commands and used|to cascade them. Then, we can execute
commands by run(), result() or iterator.cmdlet can convert corresponding types to Pipe object automatically. In above
example, range(10) is an iterator not a Pipe object. Because second item is
a Pipe object(made by pipe.filter), it turns out first item to be converted
to a Pipe object automatically.There are many useful utilities in cmdlet.cmds modules. They can provide a great
convenience to build up useful pipes. Here is an example:fromcmdlet.cmdsimport*query_topic='find ./mydoc -name "*.txt" -print'|readline(end=10)|match(r'^[tT]opic:\s*(?P<topic>.+)\s*',to=dict)|values('topic')fortopicinquery_topic:printtopicIn above example, the goal is to query topic from article files. To achieve the
goal, we have to:Search text files in a given folder.Read first 10 lines from each file.Find the line that matched 'topic: foo bar' pattern.Extract the topic string.With the utilities provided bycmdlet.cmds, we only need to write a few of
code. The first string which starts with 'find' is a normal shell script. It is
converted toshpipe automatically and executed with system shell. Thereadlinepipe can open files whose name passed from sh pipe.matchpipe
andvaluespipe work together to extract topic from file content.Above example shows not only small code but also readability. It's really easy
to understand the purpose of source code.NOTE:When using cmdlet's pipe mechanism, make sure one of yourfirst two pipe itemsis a valid Pipe object.There is another advantage to use cmdlet. The pipe object is evaluated when
calling result, run or iter. It implies you can reuse them. Let's modify
previous example.fromcmdlet.cmdsimport*# Separate from query_topic command.extract_topic=readline(end=10)|match(r'^[tT]opic:\s*(?P<topic>.+)\s*',to=dict)|values('topic')fortopicin('find ./mydoc1 -name "*.txt" -print'|extract_topic):printtopicfortopicin('find ../mydoc2 -name "*.md" -print'|extract_topic):printtopicHow to installJust like other packages on PyPI. You can use pip to download and install
automatically.$pipinstallcmdletRun piped commands and get resultThere are 3 ways to execute piped commands and get the result.Userun(cmds)orcmds.run()to execute cmds and get the last
processed data. Use this if you don't need all processed data. Or, the tasks
you need to do have been done by cascaded Pipe objects.Useresult(cmds)orcmds.result()to get the processed data in a list.
Use this method when you need to take all processed data to other mechanisms.Use cmds as aiteratorto handle the processed data one by one. It treats
cascaded Pipe objects as a pre-processing function. Use it to process data and
invoke it by a for loop to do the last processing by yourself.Function wrapperFunction should not be used in pipes directly, unless using auto-type
conversion. Cmdlet provides a set of basic wrappers to wrap function to Pipe
object.pipe.func(generator_function)The most basic wrapper. In Python, generator function is a function with yield
statement in it. The generator_function defined here is a Python generator
function with at least one argument. The first argument is a generator object
passed by previous Pipe object. generator_function can take it as input or just
leave it. It looks like:# Generator function which use prev as input.@pipe.funcdefmy_generator(prev):fordatainprev:# ... Put some code to process data ...yieldnew_data# Generator function which ignore input.@pipe.funcdefmy_generator_ignore_prev(prev):whileTrue:# ... Generate data and break loop in some conditions. ...yielddataFor example:@pipe.funcdefrandint_generator(prev,num):foriinrange(num):yieldrandom.randint(0,1000)@pipe.funcdefpower(prev,th):forninprev:yieldn**thcmds=randint_generator(10)|powerans=result(cmds)# Equals to:# ans = []# for i in range(10):#     ans.append(random.randint(0, 1000)pipe.map(function)Wrap function to a mapper. The input is a normal function with at least one
argument for data input. The returned value will be passed to next
Pipe object. It looks like:@pipe.mapdefmy_mapper(data):# ... Put some code to process data ...returnnew_dataFor example:@pipe.funcdefrandint_generator(prev,num):foriinrange(num):yieldrandom.randint(0,1000)@pipe.mapdefpower(n,th):returnn**thcmds=randint_generator(10)|powerans=result(cmds)# Equals to:# ans = []# for i in range(10):#     ans.append(random.randint(0, 1000)The power pipe can also be written in this way:power=pipe.map(lambdan,th:n**th)Anything returned by mapper will be sent to next Pipe object. If mapper return
None, next Pipe object will receive None. That is, you can't use mapper to
filter data out. That's why we have pipe.filter.pipe.filter(function)Wrap function to a filter. Filter is a function with at least one argument as
data input. Filter should return Boolean value, True or False. If True, data
from previous Pipe object is allowed to pass through. If False, data is dropped.
It looks like:@pipe.filterdefmy_filter(data):# Handle data and check conditions.ifyou_should_not_pass:returnFalseelse:returnTrueFor example:@pipe.filterdefless_than(data,thrd):returndata<thrdcmds=range(10)|less_than(3)ans=result(cmds)# Equals to:# ans = []# thrd = 3# for n in range(10):#     if n < thrd:#          ans.append()You can write filter pipe in this way:less_than=pipe.filter(lambdadata,thrd:data<thrd)pipe.reduce(function)Wrap function as a reducer. A reducer is a function which has at least two
arguments. The first one is used as accumulated result, the second one is
the data to be processed. A optional keyword argumentinitcan be used to
specify initial value to accumulated result. It looks like:@pipe.reducedefmy_reducer(accum_result,data):# Calculate new accum_result according to data.returnaccum_resultFor example:@pipe.reducedefcount_mod(accum_result,data,mod_by):if(data%mod_by)==0:returnaccum_resultelse:returnaccum_result+1cmds=range(1000)|count_mod(10,init=0)pipe.stopper(function)Wrap function as a stopper. Stopper is used to stop the pipe execution. It
returns true to stop the pipe execution. Return false to pass data to next.
It looks like:@pipe.stopperdefmy_stopper(data):ifcheck_stop_criteria(data):returnTruereturnFalseThe usage of wrapperHere is a example to show how to use function wrapper.fromrandomimportrandintfromcmdlet.cmdsimport*@pipe.funcdefrandom_number(prev,amount):foriinrange(amount):yieldrandint(0,100000)@pipe.filterdefin_range(data,lower_bound,upper_bound):returndata>=lower_boundanddata<=upper_bound@pipe.reducedefcount(accum_result,data):returnaccum_result+1@pipe.mapdefformat_output(data,format):returnformat%data# Generate 1000 random number and count how many of them between 100 and 500.# Then, format the result to 'ans=%d'.cmds=random_number(1000)|in_range(100,500)|count(init=0)|format_output('ans=%d')printcmds.run()# >>> ans=40If wrapped code is just a expression, following code shows another way to make
them:in_range=pipe.filter(lambdadata:data>=lower_boundanddata<=upper_bound)count=pipe.reduce(lambdaaccum_result,data:accum_result+1)format_output=pipe.reduce(lambdadata,format:format%data)NOTE:As you might already noticed, the number of argument using in piped commands
is different from the definition of wrapped function. You should know your
function is wrapped to a Pipe object. The function is not invoked when
cascading pipes. It is called when using run(), result() or iteration. The
arguments will be stored in Pipe object and append to the argument list of
wrapped function when it is invoked.Auto-type conversionIf the operand of|operator is not a Pipe object, cmdlet will call proper
creator to convert and wrap it to a Pipe object. The data type of operand must
be registered in cmdlet. Otherwise, exceptionUnregisteredPipeTypewill be
raised.cmdlet.cmds has registered some basic types by default. You can use them
directly.TypewrapperDescriptiontypepipe.mapConvert processed data to specified typefunctionpipe.mapWrap function as a mapper.methodpipe.mapWrap method as a mapper.tupleseqWrap tuple to generator.listseqWrap list to generator.strshWrap string to command line and execute it.unicodeshWrap string to command line and execute it.filefileobjWrap file object for read/write operation.cmdlet.cmds utilities.cmdlet.cmds has predefined some commands. Here are brief descriptions.Pipe commnds for iterable object.CommandDescriptionpackTake N elements from pipe and group them into one element.enumGenerate (index, value) pair from previous pipe.counterCount the number of data from previous pipe.flattenFlatten the data passed from previous pipe.itemsExtract (key, value) pair from a dict-like object.seqExtract any iterable object.attrExtract the value of given attribute from previous pipe.attrsExtract the value of given attributes from previous pipe.attrdictExtract the value of given attributes from previous pipe.Pipe commands for fileCommandDescriptionstdoutOutput data from previous pipe to stdout.stderrOutput data from previous pipe to stderr.readlineRead data from file line by line.fileobjRead/write file with pipe data.Pipe commands for shellCommandDescriptionshExecute system shell script to handle the stdin/stdout. The data from previous pipe will be the standard input of specified shell command.execmdExecute system shell script to handle the stdin/stdout. The data from previous pipe will be the command line to be executed.Pipe commands for stringsAlias of string methodCommandDescriptionupperalias of string.upperloweralias of string.lowercapwordsalias of string.capwordscapitalizealias of string.capitalizelstripalias of string.lstriprstripalias of string.rstripstripalias of string.stripexpandtabsalias of string.expandtabsstripalias of string.stripfindalias of string.findfmtalias of % operator of string (not string.format)rfindalias of string.rfindcountalias of string.countsplitalias of string.splitrsplitalias of string.rsplitswapcasealias of string.swapcasetranslatealias of string.translateljustalias of string.ljustrjustalias of string.rjustcenteralias of string.centerzfillalias of string.zfillreplacealias of string.replacejoinalias of string.joinsubstitutealias of string.Template.substitutesafe_substitutealias of string.Template.safe_substituteString split, search and matchCommandDescriptiongrepGrep strings with regular expression.matchGrep strings with regular expression and generate MatchObject.wildcardGrep strings with wildcard character.resplitSplit strings with regular expression.subSubstitute strings with regular expression.subnSubstitute strings with regular expression.