Package: busybee
BusyBee 🐝Simple, interactive multiprocessing for slow I/O and calculation in your notebookSimple to use as a drop-in replacement for the standardmapfunctionPrints the current progress and remaining time estimateNo external dependencies and 100% test coverageThe PyPI project page is here:https://pypi.org/project/busybee/Quick startInstall the BusyBee module viapipand use it as a replacement for your currentmapfunction. As BusyBee needs to know to total number of items the data must expose its length tolen()calls. The best approach is to provide it as alist.Install:$pip3installbusybeeCode:importbusybeeresult=busybee.map(func,data)Output:BusyBee:Startprocessing42itemswith8processes...BusyBee:1/42,2.4%(avg:3.2scpu,rem:16.5s)BusyBee:15/42,35.7%(avg:2.4scpu,rem:8.1s)BusyBee:21/42,50.0%(avg:2.5scpu,rem:6.5s)BusyBee:24/42,57.1%(avg:2.6scpu,rem:5.8s)BusyBee:34/42,81.0%(avg:2.5scpu,rem:2.5s)BusyBee:Finishedprocessing42itemsin16.1s(avg:2.6scpu)Advanced usage 👩‍💻 👨‍💻You can configure the amount of cores to be used using theprocessesargument. For this you can either provide a number (e.g. 1, 8) or a simple formula such asn/2orn-1. Thenrefers to the logical number of CPU cores returned by themultiprocessingmodule.Further, you can configure the output by providing a customstdoutsink and configuring how often you want to receive an update. You can do so by using the argumentsupdate_every_n_seconds(default: 10) andupdate_every_n_percent(default: 50).If you do not want any output, just setquiet=False.Example:importbusybeeresult=busybee.map(func,dataprocesses='n-1',update_every_n_seconds=10,update_every_n_percent=25,)Q&A 🤔Why did you built it? And why shouldn't I just use themultiprocessingmodule?I started building BusyBee when I was working with a lot of I/O and pre-processing in Python Notebooks. Parallelizing these cells made it much faster, but it was often more involved than a one-line change. For instance, the worker pool needs manual clean-up and leaks a substantial amount of memory otherwise.Also, it was hard to predict the remaining time and whether it was worth to avoid context switching or actually making some tea/coffee ☕.Is there more than themap(func, data, ...)function?Yes! While themapfunction is the most universal, there are more: Thefilter(func, data, ...)functions works similar to the regularfilterfunction applied on lists. Themk_dict(func, keys, ...)function resembles the dictionary compression syntax{k: func(k) for k in keys).I want a different output!I want to allow choosing from certain output styles. This is on my roadmap, but I do not have any certain date in mind. To maintain the simplicity I do not envision supporting custom output formatting. However, I am happy to be convinced otherwise.Contribute 👋Awesome that you are interested in improving this code! When contributing, please follow the following (common-sense) steps:Create an issue before you write any code. This allows to guide you in the right direction.If you are after a simple 1-5 line fix, you might ignore this.In the pull-request explain the high-level goal and your approach. That provides valuable context.Convince others (and yourself) that the change is safe and sound.Runpython3 -m unittest tests/test*after you added test cases for your changesRuncoverage3 run --source busybee setup.py test && coverage3 reportto ensure that the code is actually fully coveredReference/BibTex 📚If you want to reference BusyBee in documentation or articles, feel free to use this suggested BibTex snippet:@misc{hugenroth2020busybee,
  author={{Daniel Hugenroth}},
  title={BusyBee Python Software Library},
  year={2020},
  url={https://github.com/lambdapioneer/busybee},
}