Package: search-data-collector
Search Data CollectorThread-safe and atomic collection of tabular data into csv-files.IntroductionThe search-data-collector provides a single class with following methods to manage data:saveappendloadremoveThe Search-Data-Collector was created as a utility function for theGradient-Free-Optimizers- andHyperactive-package. It is intended to be used as a tool to collect search-data from the optimization run. The search-data can be collected during the optimization run as a dictionary viaappendor after the run as a dataframe with thesave-method.Theappend-method is thread-safe to work with hyperactive-multiprocessing. Thesave-method is atomic to avoid accidental data-loss, when interupting the save-process.For the Hyperactive-package the search-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search-space to convert the strings back to functions.DisclaimerThis project is in an early development stage and is sparsely tested. If you encounter bugs or have suggestions for improvements, then please open an issue.Installationpip install search-data-collectorExamplesAppend search-dataimportnumpyasnpfromhyperactiveimportHyperactivefromsearch_data_collectorimportSearchDataCollectorcollector=SearchDataCollector("./search_data.csv")# the csv is created automaticallydefparabola_function(para):loss=para["x"]*para["x"]+para["y"]*para["y"]data_dict=dict(para)# copy the parameter dictionarydata_dict["score"]=-loss# add the score to the dictionarycollector.append(data_dict)# you can append a dictionary to the csvreturn-losssearch_space={"x":list(np.arange(-10,10,0.1)),"y":list(np.arange(-10,10,0.1)),}hyper=Hyperactive()hyper.add_search(parabola_function,search_space,n_iter=1000)hyper.run()search_data=hyper.search_data(parabola_function)search_data=collector.load(search_space)# load dataprint("\nsearch_data\n",search_data)Save search-dataimportnumpyasnpfromhyperactiveimportHyperactivefromsearch_data_collectorimportSearchDataCollectorcollector=SearchDataCollector("./search_data.csv")# the csv is created automaticallydefparabola_function(para):loss=para["x"]*para["x"]+para["y"]*para["y"]return-losssearch_space={"x":list(np.arange(-10,10,0.1)),"y":list(np.arange(-10,10,0.1)),}hyper=Hyperactive()hyper.add_search(parabola_function,search_space,n_iter=1000)hyper.run()search_data=hyper.search_data(parabola_function)collector.save(search_data)# save a dataframe insteadsearch_data=collector.load(search_space)# load dataprint("\nsearch_data\n",search_data)Functions in the search-space/search-dataimportnumpyasnpfromhyperactiveimportHyperactivefromsearch_data_collectorimportSearchDataCollectorcollector=SearchDataCollector("./search_data.csv")# the csv is created automaticallydefparabola_function(para):loss=para["x"]*para["x"]+para["y"]*para["y"]return-loss# just some dummy functions to show how this worksdeffunction1():print("this is function1")deffunction2():print("this is function2")deffunction3():print("this is function3")search_space={"x":list(np.arange(-10,10,0.1)),"y":list(np.arange(-10,10,0.1)),"string.example":["string1","string2","string3"],"function.example":[function1,function2,function3],}hyper=Hyperactive()hyper.add_search(parabola_function,search_space,n_iter=30)hyper.run()search_data=hyper.search_data(parabola_function)collector.save(search_data)# save a dataframe instead of appending a dictionarysearch_data=collector.load()# load dataprint("\nIn this dataframe the 'function.example'-column contains strings, which are the '__name__' of the functions.\nsearch_data\n",search_data,"\n",)search_data=collector.load(search_space)# load data with search-spaceprint(print("\nIn this dataframe the 'function.example'-column contains the functions again.\nsearch_data\n",search_data,"\n",))