Package: rigorous-recorder
What is this?I needed an efficient data logger for my machine learning experiments. Specifically one thatcould log in a hierarchical way (not one big global logging variable)while still having a flat table-like structure for performing queries/summarieswithout having tons of duplicated dataThis library would likely work well with PySparkWhat is a Use-case Example?Lets say you're going to perform3 experimentseach experiment has 10 episodeseach episode has 100,000 timestepsthere is an anxand ayvalue at each timestepExample goal:We want to get the averagexvalue across all timesteps in episode 2 (we don't care what experiment they're from)Our timestamp data could look like:record1={"x":1,"y":1}# first timesteprecord2={"x":2,"y":2}# second timesteprecord3={"x":3,"y":3}# third timestepProblemThose records don't contain the experiment number or the episode number (and we need those for our goal)Bad SolutionDuplicating the data would provide a flat structure, but (for 100,000 timesteps) thats a huge memory costrecord1={"x":1,"y":1,"episode":1,"experiment":1,}# first timesteprecord2={"x":2,"y":2,"episode":1,"experiment":1,}# second timesteprecord3={"x":3,"y":3,"episode":1,"experiment":1,}# third timestepGood-ish SolutionWe could use references to be both more efficient and allow adding parent data after the fact# parent dataexperiment_data={"experiment":1}episode_data={"episode":1,"parent":experiment_data}record1={"x":1,"y":1,"parent":episode_data}# first timesteprecord2={"x":2,"y":2,"parent":episode_data}# second timesteprecord3={"x":3,"y":3,"parent":episode_data}# third timestepWe could reduce the cost of key duplication by having shared keys# parent dataexperiment_data={"experiment":1}episode_data={"episode":1,"parent":experiment_data}episode_keeper={"parent":episode_data}# timestep 0episode_keeper={"x":[1],"y":[1],"parent":episode_data}# first timestep (keys added on-demand)episode_keeper={"x":[1,2],"y":[1,2],"parent":episode_data}# second timestepepisode_keeper={"x":[1,2,3],"y":[1,2,3],"parent":episode_data}# third timestepHow does Rigorous Recorder Fix This?The "Good-ish Solution" above is still crude, this library cleans it upTheRecorderclass in this library is the core/pure data structureTheExperimentCollectionclass automates common boilerplate for saving (python pickle), catching errors, managing experiments, etcfromrigorous_recorderimportRecorderrecorder=Recorder()# parent dataexperiment_recorder=Recorder(experiment=1).set_parent(recorder)episode_recorder=Recorder(episode=1).set_parent(experiment_recorder)episode_recorder.push(x=1,y=1)# timestep1episode_recorder.push(x=2,y=2)# timestep2episode_recorder.push(x=3,y=3)# timestep3recorder.save_to("where/ever/you_want.pickle")How do I use this?pip install rigorous-recorderfromrigorous_recorderimportRecordKeeper,ExperimentCollectionfromstatisticsimportmeanasaveragefromrandomimportrandom,sample,choicescollection=ExperimentCollection("data/my_study")# <- this string is a filepathnumber_of_new_experiments=1for_inrange(number_of_new_experiments):# at the end (even when an error is thrown), all data is saved to disk automatically# experiment number increments based on the last saved-to-disk experiment number# running again (after error) won't double-increment the experiment number (same number until non-error run is achieved)withcollection.new_experiment()asexperiment_recorder:# we can create a hierarchy like this:##                          experiment_recorder#                           /              \#               model1_recorder           model2_recorder#                /        |                 |           \# m1_train_recorder m1_test_recorder   m2_test_recorder m2_train_recorder#model1_recorder=RecordKeeper(model="model1").set_parent(experiment_recorder)model2_recorder=RecordKeeper(model="model2").set_parent(experiment_recorder)## training#model1_train_recorder=RecordKeeper(training=True).set_parent(model1_recorder)model2_train_recorder=RecordKeeper(training=True).set_parent(model2_recorder)foreach_indexinrange(100_000):# one approachmodel1_train_recorder.push(index=each_index,loss=random())# alternative approach (same outcome)model2_train_recorder.add(index=each_index)# - this way is very handy for adding data in one method (like a loss func)#   while calling .commit() in a different method (like update weights)model2_train_recorder.add({"loss":random()})model2_train_recorder.commit()## testing#model1_test_recorder=RecordKeeper(testing=True).set_parent(model1_recorder)model2_test_recorder=RecordKeeper(testing=True).set_parent(model2_recorder)foreach_indexinrange(500):# one methodmodel1_test_recorder.push(index=each_index,accuracy=random(),)# alternative way (same outcome)model2_test_recorder.add(index=each_index,accuracy=random())model2_test_recorder.commit()### Analysis##all_records=collection.recordsprint("first record",all_records[0])# behaves just like a regular dictionary# slice across both models (first 500 training records from both models)records_first_half_of_time=tuple(eachforeachinall_recordsifeach["training"]andeach["index"]<500)# average loss across both modelsfirst_half_average_loss=average(tuple(each["loss"]foreachinrecords_first_half_of_time))# average only for model 1model1_first_half_loss=average(tuple(each["loss"]foreachinrecords_first_half_of_timeifeach["model"]=="model1"))# average only for model 2model2_first_half_loss=average(tuple(each["loss"]foreachinrecords_first_half_of_timeifeach["model"]=="model2"))What are some other details?TheExperimentCollectionadds 6 keys as a parent to every record:experiment_number     # int
error_number          # int, is only incremented for back-to-back error runs
had_error             # boolean for easy filtering
experiment_start_time # the output of time.time() from python's time module
experiment_end_time   # the output of time.time() from python's time module
experiment_duration   # the difference between start and end (for easy graphing/filtering)