Package: beam-ds
BeamDS (Beam Data Science)What is Beam for?Beam was created by data-science practitioners for data-science practitioners. It is designed as an ecosystem for developing and deploying data-driven algorithms in Python. It aims to increase productivity, efficiency, and performance in the research phase and to provide production-grade tools in the deployment part.Our Guiding PrinciplesSupport all phases of data-driven algorithm development:Data explorationData manipulation, preprocessing, and ETLs (Extract, Transform and Load)Algorithm selectionAlgorithm trainingHyperparameter tuningodel deploymentLifelong learningProduction level coding from the first line of code: no more quick and dirty Proof Of Concepts (POC). Every line of code counts toward a production model.Consume effectively all resources: use multi-core, multi-GPUs, distributed computing, remote storage solutions, and databases to enable as much as possible productivity by the resources at hand.Be agile: Development and production environments can change rapidly. Beam minimizes the friction of changing environments, filesystems, and computing resources to almost zero.Be efficient: every line of code in Beam is optimized to be as efficient as possible and to avoid unnecessary overheads.Easy to deploy and use algorithms: make deployment as easy as a line of code, import remote algorithms and services by their URI, and no more.Excel your algorithms: Beam comes with some state-of-the-art deep neural network implementations. Beam will help you store, analyze, and return to your running experiments with ease. When you are done, with development, beam will help you optimize your hyperparameters on your GPU machines.Data can be a hassle: beam can manipulate complex and nested data structures, including reading, processing, chunking, multi-processing, error handling, and writing.Be relevant: Beam is committed to staying relevant and updating towards the future of AI, adding support for Large Language Models (LLMs) and more advanced algorithms.Beam is the Swiss army knife that gets into your pocket: it is easy to install and maintain and it comes with the Beam Docker Image s.t. you can start developing and creating with zero effort even without an internet connection.InstallationTo install the full package from PyPi use:pipinstallbeam-ds[all]If you want to install only the data-science related components use:pipinstallbeam-ds[ds]To install only the LLM (Large Language Model) related components use:pipinstallbeam-ds[llm]The prerequisite packages will be installed automatically, they can be found in the setup.cfg file.Build from sourceThis BeamDS implementation follows the guide athttps://packaging.python.org/tutorials/packaging-projects/install the build package:python-mpipinstall--upgradebuildto reinstall the package after updates use:Now run this command from the same directory where pyproject.toml is located:python-mbuildreinstall the package with pip:pipinstalldist/*.whl--force-reinstallGetting StartedThere are several examples both in .py files (in the examples folder) and in jupyter notebooks (in the notebooks folder).
Specifically, you can start by looking into the beam_resources.ipynb notebook which makes you familiar the different
resources available in Beam.Go To the beam_resource.ipynb pageBuilding the Beam-DS docker imageThe docker image is based on the latest official NVIDIA pytorch image.
To build the docker image from Ubuntu host, you need to:update nvidia drivers to the latest version:https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-20-04-focal-fossa-linuxinstall docker:https://docs.docker.com/desktop/linux/install/ubuntu/Install NVIDIA container toolkit:https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#install-guideInstall and configure NVIDIA container runtime:https://stackoverflow.com/a/61737404Build the sphinx documentationFollowhttps://github.com/cimarieta/sphinx-autodoc-exampleProfiling your code with ScaleneScalene is a high-performance python profiler that supports GPU profiling.
To analyze your code with Scalene use the following arguments:scalene--reduced-profile--outfileOUTFILE.html--html---your_prog.py<youradditionalarguments>Uploading the package to PyPiInstall twine:python-mpipinstall--user--upgradetwineBuild the package:python-mbuildUpload the package:python-mtwineupload--repositorypypidist/*