Package: smallab
smallab:small_blue_diamond: :microscope: :small_blue_diamond:smallab (Small Lab) is an experiment framework designed to be easy to use with your experiment.The code in this repo should be understandable as a whole in at most 10 minutes.FeaturesEasy to understand and simpleEasy parallelization of experimentsOnly runs not previously completed experimentsHooks allow monitoring batch progressAll parameters to methods use the typing moduleDashboard for monitoring experiment progressInstallationpip install smallabor clone here from sourceDocumentationhttps://smallab.readthedocs.io/en/latest/UsageCheck outthe examples folderordemo.py(Copied here)importtypingimportrandomimportosimportdill#What you need to import from smallabfromsmallab.experiment_types.experimentimportExperimentfromsmallab.runner.runnerimportExperimentRunner#Write a simple experimentclassSimpleExperiment(Experiment):#Need to implement this method, will be passed the specification#Return a dictionary of resultsdefmain(self,specification:typing.Dict)->typing.Dict:random.seed(specification["seed"])foriinrange(specification["num_calls"]):#Advance the random number generator some amountrandom.random()if"fail"inspecificationandspecification["fail"]:raiseException()return{"number":random.random()}runner=ExperimentRunner()#Optional: Email yourself when the whole batch is done#Read https://stackoverflow.com/questions/5619914/sendmail-errno61-connection-refused about how to start an stmp serevrfromsmallab.utilities.email_hooksimportEmailCallbackBatchOnlyrunner.attach_callbacks([EmailCallbackBatchOnly("test@test.com",40)])#Take it back off since we don't actually want to bother Mr. Testrunner.attach_callbacks([])#Set the specifications for our experiments, the author reccomends reading this from a json file!specifications=[{"seed":1,"num_calls":1},{"seed":2,"num_calls":1}]#Fire off the experimentrunner.run("random_number",specifications,SimpleExperiment())#Read back our resultsforroot,_,filesinos.walk(runner.get_save_directory("random_number")):forfnameinfiles:if".pkl"infname:withopen(os.path.join(root,fname),"rb")asf:results=dill.load(f)print(results["specification"]["seed"])print(results["result"]["number"])fromsmallab.specification_generatorimportSpecificationGenerator#If you want to run a lot of experiments but not manual write out each one, use the specification generator.#Note: This is also JSON serializable, so you could store this in a json filegeneration_specification={"seed":[1,2,3,4,5,6,7,8],"num_calls":[1,2,3]}#Call the generate method. Will create the cross product.specifications=SpecificationGenerator().generate(generation_specification)print(specifications)runner.run("random_number_from_generator",specifications,SimpleExperiment(),continue_from_last_run=True)#Read back our resultsforroot,_,filesinos.walk(runner.get_save_directory("random_number_from_generator")):forfnameinfiles:if".pkl"infname:withopen(os.path.join(root,fname),"rb")asf:results=dill.load(f)print(results["specification"]["seed"])print(results["result"]["number"])#If you have an experiment you want run on a lot of computers you can use the MultiComputerGenerator#You assign each computer a number from 0..number_of_computers-1 and it gives each computer every number_of_computerth specificationfromsmallab.specification_generatorimportMultiComputerGeneratorall_specifications=SpecificationGenerator().from_json_file('test.json')g1=MultiComputerGenerator(0,2)g2=MultiComputerGenerator(1,2)specifications_1=g1.from_json_file("test.json")specifications_2=g2.from_json_file("test.json")assertlen(specifications_1)+len(specifications_2)==len(all_specifications)#Need to freeze the sets in order to do set manipulation on dictionariesspecifications_1=set([frozenset(sorted(x.items()))forxinspecifications_1])specifications_2=set([frozenset(sorted(x.items()))forxinspecifications_2])all_specifications=set([frozenset(sorted(x.items()))forxinall_specifications])#This will generate two disjoint sets of specificationsassertspecifications_1.isdisjoint(specifications_2)#That together make the whole specificationassertspecifications_1.union(specifications_2)==all_specifications#You can use the provided logging callbacks to log completion and failure of specific specifcationsfromsmallab.utilitiesimportlogger_callbacksrunner.attach_callbacks([LoggingCallback()])runner.run('with_logging',SpecificationGenerator().from_json_file("test.json"),SimpleExperiment(),continue_from_last_run=True)How it worksTheExperimentRunnerclass is passed a list of dictionaries of specifications.
These dictionaries need to be json serializable.TheExperimentRunnerlooks at thecompleted.jsonin the folder for the batch name (the name parameter of the.runmethod) and computes which experiments need to be run.
The experiments that need to run are the specifications not in thecompleted.json.TheExperimentRunnerbegins runnning the batch either in parallel or single threaded.
If the parallel implementation is used each specification isjoblib's threaded backend.Once all experiments are either completed or failed (they threw an exception) the results are saved as apicklefile.
The results are saved in a dictionary that looks like{"specification":<thespecificationtheexperimentwaspassed>,"result":<whattheexperiment.mainreturned>}The return value of the experiment.mainfunction must bepickleserializable.CallbacksThe runner has several hooks which are called at different times.on_specification_completecalled whenever a specification completes running (Ususually a single experiment)on_specification_falurecalled whenver a specification fails running (Throws an exception)on_batch_completecalled after runner.runhas finished running, passed all the succesfully completed specificationson_batch_failurecalled after runner.runhas finished running, passed all the failed specificationsFolder StructureEach experiment is saved in the following structureexperiment_runs/
  <name>/                        # The name you provide to runner.run
    <specification_hash>/        # A hash of the dictionary you provide as the specification
      specification.json         # The specification.json
        <specification_hash>.pkl # The results dictionary