Package: exp-runner
Experiment Runner (exp-runner)exp-runneris a simple and extensible framework for data analysis and machine learning experiments in Python.StructureThe framework includes following step:Data loadingData transformationModel training and testingPerformance evaluationResults savingMain featuresGenerability: Variaty of models and methods are supported and it can be used in a number of tasks
(such as preprocessing, dimensionality reduction, classification,
regression, clustering, statistical tests, etc.)Flexability: Steps can be easily skipped and/or includedDynamic loading: Automatically imports modules during runtime - no additional lines are neededInstallationpipinstallexp-runnerUsageLet's say, your project has the following structure:MyAwesomeProject/
        main.py
        my_custom_module.py

        data/
                data_00.npy
                data_01.npy
                ...
                data_NN.npy

        protocols/
                experiment_config.json

        results/Just give me a code!You just need to describe yourframeworkin theJSONconfiguration file:experiment_config.json{"Setup":{"description":"You can add detailed description of the experiment","random_seed":42},"Dataset":{"class":"my_custom_module.MyAwesomeDataLoader","args":{"path_to_data":"data/*.npy"}},"Transforms":[{"class":"sklearn.decomposition.PCA","args":{"n_components":3,"whiten":true}}],"Model":{"class":"sklearn.cluster.KMeans","args":{"n_clusters":3,"n_jobs":-1,"verbose":0}},"Metric":{"class":"my_custom_module.SklearnMetricWrapper","args":{"metric":"normalized_mutual_info_score"}},"Saver":{"class":"my_custom_module.CSVReport","args":{"path_to_output":"results/evaluation_results.csv","sep":";"}}}Here are aforementioned classes (click):my_custom_module.pyimportosimportglobimportnumpyasnpimportsklearn.metricsfromexp_runnerimportDataset,Metric,SaverfromcollectionsimportdefaultdictfromtypingimportAny,Dict,List,Union,NoReturn,Iterable,Callablefromsklearn.model_selectionimportStratifiedShuffleSplitclassMyAwesomeDataLoader(Dataset):def__init__(self,path_to_data:str,test_size:float=0.1,training:bool=True):super(MyAwesomeDataLoader,self).__init__()self._samples=dict()self._labels=dict()self._splits=defaultdict(dict)paths_to_data=glob.glob(path_to_data)forpathinpaths_to_data:fname=os.path.basename(path)data=np.load(path)X=data[:,:-1]y=data[:,-1]indices_train,indices_test=next(StratifiedShuffleSplit(test_size=test_size).split(X,y))self._samples[fname]=Xself._labels[fname]=yself._splits[fname]['train']=indices_trainself._splits[fname]['test']=indices_testself._indices=list(self._samples.keys())self._training=trainingdef__getitem__(self,index:int)->Dict[str,Dict[str,Union[str,np.ndarray]]]:ifnot(0<=index<len(self._indices)):raiseIndexErrorfname=self._indices[index]item={'X':self._samples[fname][self._splits[fname]['train']ifself.trainingelseself._splits[fname]['test']],'y':self._labels[fname][self._splits[fname]['train']ifself.trainingelseself._splits[fname]['test']]}item['desc']='it is possible to add description for each data sample'return{'filename':fname,'item':item}def__len__(self)->int:returnlen(self._indices)@propertydeftraining(self):returnself._trainingclassSklearnMetricWrapper(Metric):def__init__(self,metric:str):super(SklearnMetricWrapper,self).__init__()metric=getattr(sklearn.metrics,metric)self._metric:Callable[[Iterable[Union[float,int]],Iterable[Union[float,int]]],float]=metricdef__call__(self,y_true:Iterable[Union[float,int]],y_pred:Iterable[Union[float,int]])->float:returnself._metric(y_true,y_pred)classCSVReport(Saver):def__init__(self,path_to_output:str,sep:str=';',append:bool=True):super(CSVReport,self).__init__()self.path_to_output=path_to_outputself.sep=sepself.mode='a+'ifappendelse'w+'defsave(self,report:List[Dict[str,Any]])->NoReturn:withopen(self.path_to_output,self.mode)ascsv:forentryinreport:line=self.sep.join([entry['filename'],entry['desc'],entry['perf']])+'\n'csv.write(line)Finally, to run your experiment type in your terminal:cd/path/to/MyAwesomeProject
pythonmain.py--configprotocols/experiment_config.json