Package: easy_sge
GridSearch-like on the gridSearching good combinaisons of parameters happen quite ofter in ML-related task. I wrote a tool that help to achieve that. If you worked with the unix system callfork, you will see a similar approach with the API.Installationpip install easy_sgeHello World exampleIn this example we wrote all combinaisons of hello/hi world/Idiap.fromeasy_sge.experiences_managerimportExperienceBaseif__name__=='__main__':ifExperienceBase.is_in_SGE():# Write your taskparameters=ExperienceBase.load_only_current()first=parameters['first']second=parameters['second']print('{}{}'.format(first,second))else:# Configure your taskmy_experience=ExperienceBase()my_experience.add_experience_key_values('first',['hello','hi'])my_experience.add_experience_key_values('second',['world','idiap'])my_experience.add_grid_parameter('-P','devel')my_experience.add_grid_parameter('-cwd','')my_experience.add_grid_parameter('-N','hello_world')my_experience.run(sync=True)Run the examplepython3 -m examples.hello_worldthen check the results after running the script (and waiting a little bit)ls -1 output/hello_world.o*output/hello_world.o5439319.1
output/hello_world.o5439319.2
output/hello_world.o5439319.3
output/hello_world.o5439319.4Each file contains the result of a combinaison. Let's check the content of the first onecat output/hello_world.o5439319.1hello worldGet all results at oncecat output/hello_world.o*hello world
hi world
hello idiap
hi idiapFizzBuzz exampleWe can also use task over a single parameter, like in fizzbuzz, which depend ofn. Here an example with fizzbuzz from [1, 100]fromeasy_sge.experiences_managerimportExperienceBaseif__name__=='__main__':ifExperienceBase.is_in_SGE():# Write your taskparameters=ExperienceBase.load_only_current()n=parameters['n']my_str=''ifn%3==0:my_str+='Fizz'ifn%5==0:my_str+='Buzz'ifmy_str=='':my_str=str(n)print('(n={:04d}) ->{}'.format(n,my_str))else:# Configure your taskmy_experience=ExperienceBase()my_experience.add_experience_key_values('n',list(range(1,101)))my_experience.add_grid_parameter('-P','devel')my_experience.add_grid_parameter('-cwd','')my_experience.add_grid_parameter('-N','fizzbuzz')my_experience.run(sync=True)Run the examplepython3 -m examples.fizzbuzzGet all results at once and sort by ncat output/fizzbuzz.o* | sort(n=0001) -> 1
(n=0002) -> 2
(n=0003) -> Fizz
(n=0004) -> 4
(n=0005) -> Buzz
(n=0006) -> Fizz
(n=0007) -> 7
(n=0008) -> 8
(n=0009) -> Fizz
(n=0010) -> Buzz
(n=0011) -> 11
...
(n=0090) -> FizzBuzz
(n=0091) -> 91
(n=0092) -> 92
(n=0093) -> Fizz
(n=0094) -> 94
(n=0095) -> Buzz
(n=0096) -> Fizz
(n=0097) -> 97
(n=0098) -> 98
(n=0099) -> Fizz
(n=0100) -> BuzzUsing with conda environmentIf you want to use the same conda environmenent than the experience runner, then just letpython_executableparameter empty.Otherwise you can use conda run with the python_executable parametermy_experience=ExperienceBase(python_executable="/idiap/user/wdroz/conda_stuff/miniconda3/condabin/conda run -n rise-baseline python3")you need to specify the full path of condaDevFlitPlease installflitThen runflit install --extras devFrom pipYou can alternatively install the dev environnement from pip withpip install "easy_sge[dev]"Install pre-commitPlease install the pre-commit of the project withpre-commitinstalland for commit-msgpre-commitinstall--hook-typecommit-msgRunning testspython-mpytest