Package: holdouts-generator
Simple python package to generate and cache both random and chromosomal holdouts with arbitrary depth.How do I install this package?As usual, just download it using pip:pipinstallholdouts_generatorTests CoverageSince some software handling coverages sometime get slightly different results, here’s three of them:Generating random holdoutsSuppose you want to generate 3 layers of holdouts, respectively with 0.3, 0.2 and 0.1 as test size and 5, 3 and  2 as quantity:fromholdouts_generatorimportholdouts_generator,random_holdoutsdataset=pd.read_csv("path/to/my/dataset.csv")generator=holdouts_generator(dataset,holdouts=random_holdouts([0.3,0.2,0.1],[5,3,2]))for(training,testing),inner_holdoutsingenerator():for(inner_train,inner_test),small_holdoutsininner_holdouts():for(small_train,small_test),_insmall_holdouts():#do what you need :)Generating balanced random holdoutsSuppose you want to generate 3 layers of holdouts, as above, but now you want to enforce to apply the same proportions for each class.
In this setup, it is of foundamental importance to pass the list of classes as the last argument.fromholdouts_generatorimportholdouts_generator,balanced_random_holdoutsdataset=pd.read_csv("path/to/my/dataset.csv")classes=pd.read_csv("path/to/my/classes.csv")generator=holdouts_generator(dataset,classes,holdouts=balanced_random_holdouts([0.3,0.2,0.1],[5,3,2]))for(training,testing),inner_holdoutsingenerator():for(inner_train,inner_test),small_holdoutsininner_holdouts():for(small_train,small_test),_insmall_holdouts():#do what you need :)Generating chromosomal holdoutsSuppose you want to generate 2 layers of holdouts, two outer ones with chromosomes 17 and 18 and 3 inner ones, with chromosomes 17/18, 20 and 21:fromholdouts_generatorimportholdouts_generator,chromosomal_holdoutsdataset=pd.read_csv("path/to/my/genomic_dataset.csv")generator=holdouts_generator(dataset,holdouts=chromosomal_holdouts([([17],[([18],None),([20],None),([21],None)])([18],[([17],None),([20],None),([21],None)])]))for(training,testing),inner_holdoutsingenerator():for(inner_train,inner_test),_ininner_holdouts():#do what you need :)Generating cached holdoutsTo generate a cached holdout you just need to import instead of holdouts_generator the other method called cached_holdouts_generator.
Everything else stays basically the same, except you receive also the holdout cached key for storing the results.fromholdouts_generatorimportcached_holdouts_generator,balanced_random_holdoutsdataset=pd.read_csv("path/to/my/dataset.csv")classes=pd.read_csv("path/to/my/classes.csv")generator=cached_holdouts_generator(dataset,classes,holdouts=balanced_random_holdouts([0.3,0.2],[5,3]))for(training,testing),key,inner_holdoutsingenerator():for(inner_train,inner_test),inner_key,small_holdoutsininner_holdouts():#do what you need :)Clearing the holdouts cacheJust run the methodclear_cache:fromholdouts_generatorimportclear_cacheclear_cache(cache_dir=".holdouts"# This is the default cache directory)Clearing the invalid holdoutsSometimes it can happen that by moving around holdouts or
simply by running parallel processes on clusters with machine with different specifics
some holdouts can be created twice, overriding the original cache.In this unlikely scenario, the holdouts will be marked astempered.
To delete these holdouts use the following:fromholdouts_generatorimportclear_invalid_cacheclear_invalid_cache(cache_dir=".holdouts"# This is the default cache directory)Clearing the invalid resultsAs you can get invalid holdouts, it is also possible to get invalid results that map
to invalid holduts. For this reason there is a method to delete these results:fromholdouts_generatorimportclear_invalid_resultsclear_invalid_results(results_directory:str="results",# This is the default results directorycache_dir=".holdouts"# This is the default cache directory)