Package: entropic
EntropicFrom chaos, information.Entropic is a data pipeline framework designed to provide scientists with a simple and efficient way to access data from their experiments.
You can findthe full documentation here.RequirementsEntropic needs Python 3.9+, and relies mostly on:Pydanticfor data validation.Pandasfor data analysis.InstallationYou can install Entropic usingpip:pipinstallentropicUsageExampleThe most basic data pipeline that can be created with entropic consists of aPipelinesubclass which defines the directories containing the experiment results and a function that will be used to read each result file and create a pandas DataFrame from it:importpandasaspdfromentropic.processimportPipelinefromentropicimportresultsclassProcess(Pipeline):source_paths=["experiments/iteration_1","experiments/iteration_2"]extract_with=pd.read_csvp=Process()p.run()if__name__=="__main__":foriterationinresults.all:forsampleiniteration.samples:print(sample.data.raw.head())The main parts from this example are:Define your data processing class by inheriting from Pipeline:classProcess(Pipeline):source_paths=["experiments/iteration_1","experiments/iteration_2"]extract_with=pd.read_csvThesource_pathsvariable points to folders which contain the results for an iteration. Within entropic, an iteration can be thought as a set of initial conditions for which you performed an experiment and took various samples with various results.extract_withdefines a function that will read through all of the sample files and create a DataFrame from it. In this example I'm usingpandas.read_csv, but it can be any function you want -you can even custom define it and pass it toextract_with.Instantiate and run the pipeline:p=Process()p.run()Access your results using theresultsAPI:if__name__=="__main__":foriterationinresults.all:forsampleiniteration.samples:print(sample.data.raw.head())In this example the accessing of results happens on the same file in which you run the pipeline. However, for performance reasons you might want to consider splitting the processing and the analysis on two different files. In this case you only need to run the processing part once, and your data will be loaded to a JSON-based database.Example upgradeA more realistic example will involve custom iterations and samples, which need custom logic for extracting, transforming or loading them into the database.importpandasaspdfromentropicimportresultsfromentropic.sourcesimportBaseSample,Iterationfromentropic.processimportPipelinefromentropic.sources.fieldsimportDataSourceclassKinematicSample(BaseSample):data:DataSourcespeed:float=0points_in_data:int=0classKinematicExperiment(Iteration):average_speed:float=0sample=KinematicSampleclassProcess(Pipeline):source_paths=["experiments/initial_condition_1"]iteration=KinematicExperimentdefextract(self,source_path):iteration=self.get_iteration_by_path(source_path)forfile_pathinself.get_files_from_path(source_path):raw=pd.read_csv(file_path)data_source=DataSource(file_path=file_path,raw=raw)sample=self.get_sample()(data=data_source,points_in_data=raw.shape[0])iteration.upsert_sample(sample)returniterationdeftransform(self,iteration):average=0forsampleiniteration.samples:sample.speed=(sample.data.raw["x"]/sample.data.raw["t"]).mean()average+=sample.speediteration.average_speed=average/len(iteration.samples)p=Process()p.run()results.set_iteration(KinematicExperiment)if__name__=="__main__":foriterationinresults.all:print(f"Iteration average speed={iteration.average_speed}")fori,sampleinenumerate(iteration.samples):print(f"Sample{i+1}")print(f"speed={sample.speed}")print(f"rows={sample.points_in_data}")print()A few changes have been done from the previous example:Custom iteration and sample classes were created:classKinematicSample(BaseSample):data:DataSourcespeed:float=0points_in_data:int=0classKinematicExperiment(Iteration):average_speed:float=0sample=KinematicSampleInstead of defining anextract_withfunction, the extract function is defined instead. Also, calculations can be performed on a given iteration using thetransformfunction:classProcess(Pipeline):source_paths=["experiments/initial_condition_1"]iteration=KinematicExperimentdefextract(self,file_path):raw=pd.read_csv(file_path)data_source=DataSource(file_path=file_path,raw=raw)returnself.get_sample()(data=data_source,points_in_data=raw.shape[0])deftransform(self,iteration):average=0forsampleiniteration.samples:sample.speed=(sample.data.raw["x"]/sample.data.raw["t"]).mean()average+=sample.speediteration.average_speed=average/len(iteration.samples)Note thatKinematicExperimentis being defined as the iteration for theProcessclass. You can access theiterationandsampleusingself.get_iteration()andself.get_sample(). Don't try to accessself.iterationandself.sample, as it might break!In order to properly display results, the custom iteration has to be "added" to the results API:results.set_iteration(KinematicExperiment)