Package: giving
giving — the reactive loggerDocumentationgivingis a simple, magical library that lets you log or "give" arbitrary data throughout a program and then process it as an event stream. You can use it to log to the terminal, towandbormlflow, to compute minimums, maximums, rolling means, etc., separate from your program's core logic.Inside your code,give()every object or datum that you may want to log or compute metrics about.Wrap your main loop withgiven()and define pipelines to map, filter and reduce the data you gave.ExamplesCodeOutputSimple logging# All calls to give() will log to the configured consolewithgiven().display():a,b=10,20# Without parameters: last expression + resultgive()# With parameters:# parameter is just value: value => value# parameter is key and value: key => valuegive(a*b,c=30)a: 10; b: 20
a * b: 200; c: 30Extract values into a list# give(key=value) with key == "s" will add value to `results`withgiven()["s"].values()asresults:s=0foriinrange(5):s+=igive(s)print(results)[0, 1, 3, 6, 10]Reductions (min, max, count, etc.)defcollatz(n):whilen!=1:give(n)n=(3*n+1)ifn%2else(n//2)withgiven()asgv:gv["n"].max().print("max:{}")gv["n"].count().print("steps:{}")collatz(2021)max: 6064
steps: 63Using theevalmethod instead ofwith:st,=given()["n"].count().eval(collatz,2021)print(st)63Thekscanmethodwithgiven()asgv:gv.kscan().display()give(elk=1)give(rabbit=2)give(elk=3,wolf=4)elk: 1
elk: 1; rabbit: 2
elk: 3; rabbit: 2; wolf: 4Thethrottlemethodwithgiven()asgv:gv.throttle(1).display()foriinrange(50):give(i)time.sleep(0.1)i: 0
i: 10
i: 20
i: 30
i: 40The above examples only show a small number ofall the available operators.GiveThere are multiple ways you can usegive.givereturns Noneunlessit is given a single positional argument, in which case it returns the value of that argument.give(key=value)This is the most straightforward way to usegive: you write out both the key and the value associated.Returns:Nonex = give(value)When no key is given, but the result ofgiveis assigned to a variable, the key is the name of that variable. In other words, the above is equivalent togive(x=value).Returns:The valuegive(x)When no key is given and the result isnotassigned to a variable,give(x)is equivalent togive(x=x). If the argument is an expression likex * x, the key will be the string"x * x".Returns:The valuegive(x, y, z)Multiple arguments can be given. The above is equivalent togive(x=x, y=y, z=z).Returns:Nonex = value; give()Ifgivehas no arguments at all, it will look at the immediately previous statement and infer what you mean. The above is equivalent tox = value; give(x=value).Returns:NoneImportant functions and methodsprintanddisplay: for printing out the streamvaluesandaccum: for accumulating into a listsubscribeandksubscribe: perform a task on every elementwhere,where_any,keep,gv["key"],gv["?key"]: filter based on keysSee here for more details.Operator summaryNot all operators are listed here.See herefor the complete list.Filteringfilter: filter with a functionkfilter: filter with a function (keyword arguments)where: filter based on keys and simple conditionswhere_any: filter based on keyskeep: filter based on keys (+drop the rest)distinct: only emit distinct elementsnorepeat: only emit distinct consecutive elementsfirst: only emit the first elementlast: only emit the last elementtake: only emit the first n elementstake_last: only emit the last n elementsskip: suppress the first n elementsskip_last: suppress the last n elementsMappingmap: map with a functionkmap: map with a function (keyword arguments)augment: add extra keys using a mapping functiongetitem: extract value for a specific keysole: extract value from dict of length 1as_: wrap as a dictReductionreduce: reduce with a functionscan: emit a result at each reduction steproll: reduce using overlapping windowskmerge: merge all dictionaries in the streamkscan: incremental version ofkmergeArithmetic reductionsMost of these reductions can be called with thescanargument set toTrueto usescaninstead ofreduce.scancan also be set to an integer, in which caserollis used.averageaverage_and_variancecountmaxminsumvarianceWrappingwrap: give a special key at the beginning and end of a blockwrap_inherit: give a special key at the beginning and end of a blockinherit: add default key/values for every give() in the blockwrap: plug a context manager at the location of agive.wrapkwrap: same as wrap, but pass kwargsTimingdebounce: suppress events that are too close in timesample: sample an element every n secondsthrottle: emit at most once every n secondsDebuggingbreakpoint: set a breakpoint whenever data comes in. Use this with filters.tag: assigns a special word to every entry. Use withbreakword.breakword: set a breakpoint on a specific word set bytag, using theBREAKWORDenvironment variable.Otheraccum: accumulate into a listdisplay: print out the stream (pretty).print: print out the stream.values: accumulate into a list (context manager)subscribe: run a task on every elementksubscribe: run a task on every element (keyword arguments)ML ideasHere are some ideas for using giving in a machine learning model training context:fromgivingimportgive,givendefmain():model=Model()foriinrange(niters):# Give the model. give looks at the argument string, so# give(model) is equivalent to give(model=model)give(model)loss=model.step()# Give the iteration number and the loss (equivalent to give(i=i, loss=loss))give(i,loss)# Give the final model. The final=True key is there so we can filter on it.give(model,final=True)if__name__=="__main__":withgiven()asgv:# ===========================================================# Define our pipeline **before** running main()# ===========================================================# Filter all the lines that have the "loss" key# NOTE: Same as gv.filter(lambda values: "loss" in values)losses=gv.where("loss")# Print the losses on stdoutlosses.display()# alwayslosses.throttle(1).display()# OR: once every secondlosses.slice(step=10).display()# OR: every 10th loss# Log the losses (and indexes i) with wandb# >> is shorthand for .subscribe()losses>>wandb.log# Print the minimum loss at the endlosses["loss"].min().print("Minimum loss:{}")# Print the mean of the last 100 losses# * affix adds columns, so we will display i, loss and meanloss together# * The scan argument outputs the mean incrementally# * It's important that each affixed column has the same length as#   the losses stream (or "table")losses.affix(meanloss=losses["loss"].mean(scan=100)).display()# Store all the losses in a listlosslist=losses["loss"].accum()# Set a breakpoint whenever the loss is nan or infinitelosses["loss"].filter(lambdaloss:notmath.isfinite(loss)).breakpoint()# Filter all the lines that have the "model" key:models=gv.where("model")# Write a checkpoint of the model at most once every 30 minutesmodels["model"].throttle(30*60).subscribe(lambdamodel:model.checkpoint())# Watch with wandb, but only once at the very beginningmodels["model"].first()>>wandb.watch# Write the final model (you could also use models.last())models.where(final=True)["model"].subscribe(lambdamodel:model.save())# ===========================================================# Finally, execute the code. All the pipelines we defined above# will proceed as we give data.# ===========================================================main()