Package: pensieve
PensievePensieveis a Python library for organizing objects and dependencies in a graph structure."One simply siphons the excess thoughts from one's mind, pours them into the basin, and examines them at one's leisure. It becomes easier to spot patterns and links, you understand, when they are in this form."—Albus Dumbledore(Harry Potter and the Goblet of Fire by J. K. Rowling)Pensieve for DataInJ. K. Rowling'swords:
"a witch or wizard canextracttheir own or another's memories,storethem in thePensieve,
andreviewthem later. It alsorelieves the mindwhen it becomes cluttered with information.
Anyone canexaminethe memories in the Pensieve, which also allows viewers to fully immerse
themselves in the memories".Dealing with data during data wrangling and model generation in data science is like dealing with memories
except that there is a lot more of back and forth and iteration when dealing with data.
You constantly update parameters of your models, improve your data wrangling,
and make changes to the ways you visualize or store data.
As with most processes in data science, each step along the way may take a long time to finish
which forces you to avoid rerunning everything from scratch; this approach is very error-prone as some
of the processes depend on others. To solve this problem I came up with the idea of aComputation Graphwhere the nodes represent data objects and the direction of edges indicate the dependency between them.After using Pensieve for some time myself, I have found it to be beneficial in several ways:error reduction, especially for data wrangling and model creationdata object organizationeasy transfer of datacoherent data processing and data pipelinesdata and model reproducibilityparallel processingperformance and cost analysis in terms of computation time and memory usagegraphical visualization of data and processesmost important of all:relieving the mindUsing pensieve is similar to using a dictionary:frompensieveimportPensievefrommathimportpi# initiate a pensievepensieve=Pensieve()# store a "memory" (with 1 as its content)pensieve['radius']=5# create a new memory made up of a precursor memory# it is as easy as passing a defined function or a lambda to pensievepensieve['circumference']=lambdaradius:2*pi*radiusprint(pensieve['circumference'])outputs:31.41592653589793Changing the radius, in this example, will affect the circumference
but it is only calculated when needed:pensieve['radius']=6print(pensieve['circumference'])outputs37.69911184307752InstallationpipinstallpensieveUsagePensieve storesmemoriesandfunctionsthat define the relationship between memories.ConceptsMemoryAPensieveis acomputation graphwhere the nodes hold values and edges
show dependency between nodes. Each node is called aMemory.Everymemoryhas two important attributes:key: the name of the memory which should be identicalcontent: the object the memory holdsSome memories have two other attributes:precursors: other memories a memory depends onfunction: a function that defines the relationship between a memory
and its precursorsThere are two types of memories:independentmemories (without precursors)dependentmemories (with precursors)Storing a MemoryAs explained above, you can work with pensieve similar to how you use a
dictionary. Adding a new item,i.e., a memory and its content, to pensieve is
calledstoring. In fact thePensieveclass has astoremethod which
can be used for storing new memories. However, we only use it for advanced
functionality. We do not use it as frequently because a new simpler notation
introduced since version 2 makes working pensieve much more coherent.
We will explain thestoremethod and its notation in theAdvanced Usagesection.Retrieving a MemoryRetrieving the content of a memory is like getting an item from a dictionary.print(pensieve['circumference'])Independent MemoriesAn independent memory is like a root node in pensieve. It holds an object and
it does not depend on any other memory.frompensieveimportPensievepensieve=Pensieve()pensieve['text']='Hello World!'pensieve['number']=1pensieve['list_of_numbers']=[1,3,2]In the above example,text,number, andlistare the names of three
independent memories and their contents are
the string'Hello World',
the integer1,
and a list consisting of three integers.Dependent Memories and PrecursorsA dependent memory is created from running afunctionon other dependent or
independent memories as the function's arguments. We call those memories,precursors;i.e., if a memory depends on another memory, the former is adependentmemory
and the latter is itsprecursor.The easiest way to define a dependent memory is by passing a function to pensieve
whose arguments match the names of precursors.defprint_and_return_first_word(text):words=text.split()print(words[0])returnwords[0]pensieve['first_word']=print_and_return_first_wordIn the above example, theprint_and_return_first_wordfunction accepts one argument:textwhich is the name of the precursor.You can also use a lambda, when possible, to define a dependent memory.pensieve['sorted_list']=lambdalist_of_numbers:sorted(list_of_numbers)SuccessorsMemories that depend on a memory are itssuccessors. If a precursor is like a
parent, a successor is like a child.In the above example,sorted_listis a successor oflist_of_numbers.StalenessIf one or more precursors of a memory change, the memory and all its successors becomesstale.
A stale memory is only refreshed when needed and if after calculation, it is found out
that the content has not changed, the successors go back to being up-to-date, but if
the content has in fact changed, the stay stale and will be updated when needed.Note: if a memory is stale, retrieving its content will update it.VisualizationfrompensieveimportPensievefrompandasimportDataFrame,concatfromnumpy.randomimportrandint,seed# set seed for the randint functionseed(17)# set up a pensieve with a top-bottom (tb) representation# the top-bottom graph_direction is purely aesthetic# you can also use lr for left to right or rl for right to left or bottom-toppensieve=Pensieve(graph_direction='tb')# choose the number of columns for two dataframespensieve['number_of_columns']=9# create generic names for the columns, in this case x_1, x_2, ...pensieve['column_names']=lambdanumber_of_columns:[f'x_{i+1}'foriinrange(number_of_columns)]# choose the range of random values, and store them as a dictionarypensieve['value_range']={'low':1,'high':5}# define a function that creates a dataframe with the above parametersdefcreate_dataframe(column_names,value_range,number_of_rows):returnDataFrame({column:randint(low=value_range['low'],high=value_range['high'],size=number_of_rows)forcolumnincolumn_names})# create the first dataframepensieve['data_1']=lambdacolumn_names,value_range:create_dataframe(column_names=column_names,value_range=value_range,number_of_rows=5)# create the second dataframepensieve['data_2']=lambdacolumn_names,value_range:create_dataframe(column_names=column_names,value_range=value_range,number_of_rows=3)# concatenate the two dataframespensieve['data_1_and_2']=lambdadata_1,data_2:concat(objs=[data_1,data_2],sort=False)# choose a coefficient for a future multiplicationpensieve['coefficient']=5# define a function that sums all the values in each row and# multiplies the result by the coefficientdefsum_and_multiply(data_1_and_2,coefficient):data=data_1_and_2.copy()data['summation']=data.apply(sum,axis=1)data['coefficient']=coefficientdata['y']=data['summation']*data['coefficient']returndata# get the result of the sum_and_multiply functionpensieve['result']=sum_and_multiply# display the pensievedisplay(pensieve)# or simply pensieve at the end of a jupyter notebook cellAdvanced UsageParallel ProcessingfrompensieveimportPensievefromtimeimportsleepfromdatetimeimportdatetime# as in other libraries, num_threads=-1 means# using as many threads as availablestart_time=datetime.now()pensieve=Pensieve(num_threads=-1,evaluate=False)pensieve['x']=1pensieve['y']=10pensieve['z']=2pensieve['w']=20defadd_with_delay(x,y):print(f'adding{x}and{y}, slowly, at{datetime.now()}')sleep(1)returnx+ypensieve['x_plus_y']=add_with_delaypensieve['z_plus_w']=lambdaz,w:add_with_delay(x=z,y=w)# we had to use a lambda for this one because the arguments# of the add_with_delay function are differentpensieve['all_the_four']=lambdax_plus_y,z_plus_w:add_with_delay(x=x_plus_y,y=z_plus_w)elapsed=datetime.now()-start_timeprint('Nothing has been calculated yet. Elapsed time:',elapsed)print('Getting all_the_four forces the calculation of everything')start_time=datetime.now()print('Result of adding the four numbers:',pensieve['all_the_four'])elapsed=datetime.now()-start_timeprint('Elapsed time:',elapsed)The above code produces the following output:Nothing has been calculated yet. Elapsed time: 0:00:00.000716
Getting all_the_four forces the calculation of everything
adding 2 and 20, slowly, at 2019-12-15 21:33:55.063888
adding 1 and 10, slowly, at 2019-12-15 21:33:55.064526
adding 11 and 22, slowly, at 2019-12-15 21:33:56.188258
Result of adding the four numbers: 33
Elapsed time: 0:00:02.341677Two of the calculations were executed in parallel:x + yandz + w.
With an overhead of0.34seconds, the three calculations took2.34seconds.Let's see what happens if we do it the ordinary way:start_time=datetime.now()x=1y=10z=2w=20x_plus_y=add_with_delay(x,y)z_plus_w=add_with_delay(z,w)all_the_four=add_with_delay(x_plus_y,z_plus_w)print('Result of adding the four numbers:',all_the_four)elapsed=datetime.now()-start_timeprint('Elapsed time:',elapsed)This time the following output is produced:adding 1 and 10, slowly, at 2019-12-15 21:38:11.618910
adding 2 and 20, slowly, at 2019-12-15 21:38:12.620105
adding 11 and 22, slowly, at 2019-12-15 21:38:13.625195
Result of adding the four numbers: 33
Elapsed time: 0:00:03.011291With an overhead of0.01seconds, the three calculations
ran one after the other and took3.01seconds.ThestoreMethodTBD