Package: igogo
igogo 🐎🏎️Execute several jupyter cells at the same timeHave you ever just sited and watched a long-running jupyter cell?Now, you can continue to work in the same notebook freelyhttps://user-images.githubusercontent.com/25539425/227176976-2bdda463-ecc9-4431-afec-6d31fbd4c214.movUse CasesYou have a long-running cell, and you need to check something.
You can just start the second cell without interrupting a long-running cell.Example:you run a machine learning train loop and want to immediately save the model's weights or check metrics.
Withigogoyou can do so without interrupting the training.If you need to compare the score of some function with different parameters, you can run several
functions at the same time and monitor results.Example:you have several sets of hyperparameters and want to compare them.
You can start training two models, monitoring two loss graphs at the same time.Process data in chunks. Check processed data for validityExample:you do data processing in steps. Withigogoyou can execute several steps at the same time
and process data from the first processing step in the second processing step in chunks.
Also, you can quickly check that the first step produces the correct resultsInstallIgogo is available through PyPi:pipinstalligogoWait, isn't it just a background job? No.No multithreading, no data races, no locks.
You can freely operate with your notebook variables without the risk of corrupting them.Beautiful output. When several cells execute in parallel,
all printed data is displayed in the corresponding cell's output. No more twisted and messed out concurrent outputs.Easily cancel jobs, wait for completion, and start the new ones.Control execution of jobs through widgets.UsageAt the core of igogo is collaborative execution. Jobs need to explicitly allow other jobs to execute throughigogo.yielder(). Mind that regular cells also represent a job.Placingigogo.yielder()in code that is not executed in igogo job is not a mistake. It will return immediately. So, you don't need to care about keepingigogo.yielder()only in igogo jobs. You can place it anywhereTo start an igogo job, you can use%%igogocell magic or function decorator.importigogo@igogo.jobdefhello_world(name):foriinrange(3):print("Hello, world from",name)# allows other jobs to run while asleep# also can be `igogo.yielder()`igogo.sleep(1)returnnameCall function as usual to start a job:hello_world('igogo'),hello_world('other igogo');https://user-images.githubusercontent.com/25539425/227186815-6870e348-46e6-4086-a89b-be416c0cc1a7.movConfigure JobsDecorator@igogo.jobhas several useful parameters.kindAllows to set how to render output. Possible options:text,markdown,htmlDefault:textdisplaysAs igogo job modify already executed cell, it needs to have spare placeholders for rich output.
This parameter specifies how many spare displays to spawn. Default:1nameUser-friendly name of igogo job.warn_rewriteShould warn rewriting older displays? Default:Trueauto_display_figuresShould display pyplot figures created inside igogo automatically? Default:TrueMarkdown example:https://user-images.githubusercontent.com/25539425/227203729-af94582c-8fe2-40fe-a6f0-6489a374a88f.movDisplay Additional DataPyplot figures will be automatically displayed in igogo cell.You can also useigogo.displayinside a job to display any other content or several figures. Mind that displays must be pre-allocated by specifying displays number inigogo.job(displays=...)importnumpyasnpimportmatplotlib.pyplotaspltimportigogodefexperiment(name,f,i):x=np.linspace(0,i/10,100)fig=plt.figure()plt.plot(x,f(x))plt.gca().set_title(name)igogo.display(fig)fig=plt.figure()plt.scatter(x,f(x))plt.gca().set_title(name)igogo.display(fig)igogo.sleep(0.05)As noted in "Configure jobs" section,igogojobs have limited number of displays.
If you try to display more objects than job has, warning will be shown and the oldest displays will be overwritten.Cell MagicThe same way with%%igogo:%load_extigogo%%igogoname='igogo'foriinrange(3):print("Hello, world from",name)igogo.sleep(1)WidgetsAll executedigogojobs spawn a widget that allows to kill them. Jobs are not affected byKeyboardInterruptKilling JobsApart from killing through widgets,igogojobs can be killed programmatically.igogo.stop()Can be called insideigogojob to kill itself.igogo.stop_all()Stops all runningigogojobsigogo.stop_latest()Stops the latestigogojob. Can be executed several times.igogo.stop_by_cell_id(cell_id)Kills all jobs that were launched in cell withcell_id(aka [5], cell_id=5).Also, you can stop jobs of one specific function.hello_world.stop_all()Stops alligogojobs created byhello_world()Supported ClientsCurrently,igogoruns fully correct on:Jupyter LabJupyterRuns but has problems with output from igogo jobs. Jobs are executed, but there could be problems with widgets and output:VSCode. For some reason it does not update display data. Therefore, no output is produced.DataSpell. It displays[object Object]and not output.Colab. It does not support updating content of executed cellsMore ExamplesCheck out pretty notebooksTrain model and check metricshttps://user-images.githubusercontent.com/25539425/227651626-cba8a317-a986-4971-9639-84cdb388e2d3.movAlso, you can modify training parameters, freeze/unfreeze layers, switch datasets, etc. All you need is to placeigogo.yielder()in train loop.Process data and montitor executionimportigogoimportnumpyasnpfromtqdm.autoimporttqdm%load_extigogoraw_data=np.random.randn(100000,100)result=[]defrow_processor(row):returnnp.mean(row)%%igogoforiintqdm(range(len(raw_data))):result.append(row_processor(raw_data[i]))igogo.yielder()result[-1]Process data in chunksimportigogoimportnumpyasnpfromtqdm.autoimporttqdm%load_extigogoraw_data=np.random.randn(5000000,100)igogo_yield_freq=32igogo_first_step_cache=[]result=[]%%igogoforiintqdm(range(len(raw_data))):processed=np.log(raw_data[i]*raw_data[i])igogo_first_step_cache.append(processed)ifi>0andi%igogo_yield_freq==0:igogo.yielder()# allow other jobs to execute%%igogoforiintqdm(range(len(raw_data))):whilei>=len(igogo_first_step_cache):# wait for producer to process dataigogo.yielder()result.append(np.mean(igogo_first_step_cache[i]))https://user-images.githubusercontent.com/25539425/227224077-a3ce664c-cb52-4aa2-a3fe-71ac5a03cdeb.mov