Package: kp
KomputeThe general purpose GPU compute framework for cross vendor graphics cards (AMD, Qualcomm, NVIDIA & friends)Blazing fast, mobile-enabled, asynchronous, and optimized for advanced GPU acceleration usecases.💬Join the Discord & Community Calls🔋Documentation💻Blog Post⌨Examples💾Kompute is backed by the Linux Foundation as ahosted projectby the LF AI & Data Foundation.Principles & FeaturesFlexible Python modulewithC++ SDKfor optimizationsAsynchronous & parallel processingsupport through GPU family queuesMobile enabledwith examples via Android NDK across several architecturesBYOV:Bring-your-own-Vulkan designto play nice with existing Vulkan applicationsExplicit relationships for GPU and hostmemory ownership and memory managementRobust codebase with90% unit test code coverageAdvanced use-cases onmachine learning 🤖,mobile development 📱andgame development 🎮.Active community withmonthly calls, discord chat and moreGetting StartedBelow you can find a GPU multiplication example using the C++ and Python Kompute interfaces.You canjoin the Discordfor questions / discussion, open agithub issue, or readthe documentation.Your First Kompute (C++)The C++ interface provides low level access to the native components of Kompute, enabling foradvanced optimizationsas well asextension of components.voidkompute(conststd::string&shader){// 1. Create Kompute Manager with default settings (device 0, first queue and no extensions)kp::Managermgr;// 2. Create and initialise Kompute Tensors through manager// Default tensor constructor simplifies creation of float valuesautotensorInA=mgr.tensor({2.,2.,2.});autotensorInB=mgr.tensor({1.,2.,3.});// Explicit type constructor supports uint32, int32, double, float and boolautotensorOutA=mgr.tensorT<uint32_t>({0,0,0});autotensorOutB=mgr.tensorT<uint32_t>({0,0,0});std::vector<std::shared_ptr<kp::Tensor>>params={tensorInA,tensorInB,tensorOutA,tensorOutB};// 3. Create algorithm based on shader (supports buffers & push/spec constants)kp::Workgroupworkgroup({3,1,1});std::vector<float>specConsts({2});std::vector<float>pushConstsA({2.0});std::vector<float>pushConstsB({3.0});autoalgorithm=mgr.algorithm(params,// See documentation shader section for compileSourcecompileSource(shader),workgroup,specConsts,pushConstsA);// 4. Run operation synchronously using sequencemgr.sequence()->record<kp::OpTensorSyncDevice>(params)->record<kp::OpAlgoDispatch>(algorithm)// Binds default push consts->eval()// Evaluates the two recorded operations->record<kp::OpAlgoDispatch>(algorithm,pushConstsB)// Overrides push consts->eval();// Evaluates only last recorded operation// 5. Sync results from the GPU asynchronouslyautosq=mgr.sequence();sq->evalAsync<kp::OpTensorSyncLocal>(params);// ... Do other work asynchronously whilst GPU finishessq->evalAwait();// Prints the first output which is: { 4, 8, 12 }for(constfloat&elem:tensorOutA->vector())std::cout<<elem<<"  ";// Prints the second output which is: { 10, 10, 10 }for(constfloat&elem:tensorOutB->vector())std::cout<<elem<<"  ";}// Manages / releases all CPU and GPU memory resourcesintmain(){// Define a raw string shader (or use the Kompute tools to compile to SPIRV / C++ header// files). This shader shows some of the main components including constants, buffers, etcstd::stringshader=(R"(#version 450layout (local_size_x = 1) in;// The input tensors bind index is relative to index in parameter passedlayout(set = 0, binding = 0) buffer buf_in_a { float in_a[]; };layout(set = 0, binding = 1) buffer buf_in_b { float in_b[]; };layout(set = 0, binding = 2) buffer buf_out_a { uint out_a[]; };layout(set = 0, binding = 3) buffer buf_out_b { uint out_b[]; };// Kompute supports push constants updated on dispatchlayout(push_constant) uniform PushConstants {float val;} push_const;// Kompute also supports spec constants on initalizationlayout(constant_id = 0) const float const_one = 0;void main() {uint index = gl_GlobalInvocationID.x;out_a[index] += uint( in_a[index] * in_b[index] );out_b[index] += uint( const_one * push_const.val );})");// Run the function declared above with our raw string shaderkompute(shader);}Your First Kompute (Python)ThePython packageprovides ahigh level interactive interfacethat enables for experimentation whilst ensuring high performance and fast development workflows.from.utilsimportcompile_source# using util function from python/test/utilsdefkompute(shader):# 1. Create Kompute Manager with default settings (device 0, first queue and no extensions)mgr=kp.Manager()# 2. Create and initialise Kompute Tensors through manager# Default tensor constructor simplifies creation of float valuestensor_in_a=mgr.tensor([2,2,2])tensor_in_b=mgr.tensor([1,2,3])# Explicit type constructor supports uint32, int32, double, float and booltensor_out_a=mgr.tensor_t(np.array([0,0,0],dtype=np.uint32))tensor_out_b=mgr.tensor_t(np.array([0,0,0],dtype=np.uint32))params=[tensor_in_a,tensor_in_b,tensor_out_a,tensor_out_b]# 3. Create algorithm based on shader (supports buffers & push/spec constants)workgroup=(3,1,1)spec_consts=[2]push_consts_a=[2]push_consts_b=[3]# See documentation shader section for compile_sourcespirv=compile_source(shader)algo=mgr.algorithm(params,spirv,workgroup,spec_consts,push_consts_a)# 4. Run operation synchronously using sequence(mgr.sequence().record(kp.OpTensorSyncDevice(params)).record(kp.OpAlgoDispatch(algo))# Binds default push consts provided.eval()# evaluates the two recorded ops.record(kp.OpAlgoDispatch(algo,push_consts_b))# Overrides push consts.eval())# evaluates only the last recorded op# 5. Sync results from the GPU asynchronouslysq=mgr.sequence()sq.eval_async(kp.OpTensorSyncLocal(params))# ... Do other work asynchronously whilst GPU finishessq.eval_await()# Prints the first output which is: { 4, 8, 12 }print(tensor_out_a)# Prints the first output which is: { 10, 10, 10 }print(tensor_out_b)if__name__=="__main__":# Define a raw string shader (or use the Kompute tools to compile to SPIRV / C++ header# files). This shader shows some of the main components including constants, buffers, etcshader="""#version 450layout (local_size_x = 1) in;// The input tensors bind index is relative to index in parameter passedlayout(set = 0, binding = 0) buffer buf_in_a { float in_a[]; };layout(set = 0, binding = 1) buffer buf_in_b { float in_b[]; };layout(set = 0, binding = 2) buffer buf_out_a { uint out_a[]; };layout(set = 0, binding = 3) buffer buf_out_b { uint out_b[]; };// Kompute supports push constants updated on dispatchlayout(push_constant) uniform PushConstants {float val;} push_const;// Kompute also supports spec constants on initalizationlayout(constant_id = 0) const float const_one = 0;void main() {uint index = gl_GlobalInvocationID.x;out_a[index] += uint( in_a[index] * in_b[index] );out_b[index] += uint( const_one * push_const.val );}"""kompute(shader)Interactive Notebooks & Hands on VideosYou are able to try out the interactive Colab Notebooks which allow you to use a free GPU. The available examples are the Python and C++ examples below:Try the interactiveC++ ColabfromBlog PostTry the interactivePython ColabfromBlog PostYou can also check out the two following talks presented at the FOSDEM 2021 conference.Both videos have timestamps which will allow you to skip to the most relevant section for you - the intro & motivations for both is almost the same so you can skip to the more specific content.Watch the video forC++ EnthusiastsWatch the video forPython & Machine LearningEnthusiastsArchitectural OverviewThe core architecture of Kompute includes the following:Kompute Manager- Base orchestrator which creates and manages device and child componentsKompute Sequence- Container of operations that can be sent to GPU as batchKompute Operation (Base)- Base class from which all operations inheritKompute Tensor- Tensor structured data used in GPU operationsKompute Algorithm- Abstraction for (shader) logic executed in the GPUTo see a full breakdown you can read further in theC++ Class Reference.Full ArchitectureSimplified Kompute Components(very tiny, check thefull reference diagram in docs for details)Asynchronous and Parallel OperationsKompute provides flexibility to run operations in an asynrchonous way through vk::Fences. Furthermore, Kompute enables for explicit allocation of queues, which allow for parallel execution of operations across queue families.The image below provides an intuition on how Kompute Sequences can be allocated to different queues to enable parallel execution based on hardware. You can see thehands on example, as well as thedetailed documentation pagedescribing how it would work using an NVIDIA 1650 as an example.Mobile EnabledKompute has been optimized to work in mobile environments. Thebuild systemenables for dynamic loading of the Vulkan shared library for Android environments, together with a workingAndroid NDK wrapperfor the CPP headers.For a full deep dive you can read the blog post "Supercharging your Mobile Apps with On-Device GPU Accelerated Machine Learning".You can also access theend-to-end example codein the repository, which can be run using android studio.More examplesSimple examplesSimple multiplication exampleRecord batch commands with a Kompute SequenceRun Asynchronous OperationsRun Parallel Operations Across Multiple GPU QueuesCreate your custom Kompute OperationsImplementing logistic regression from scratchEnd-to-end examplesMachine Learning Logistic Regression ImplementationParallelizing GPU-intensive Workloads via Multi-Queue OperationsAndroid NDK Mobile Kompute ML ApplicationGame Development Kompute ML in Godot EnginePython PackageBesides the C++ core SDK you can also use the Python package of Kompute, which exposes the same core functionality, and supports interoperability with Python objects like Lists, Numpy Arrays, etc.The only dependencies are Python 3.5+ and Cmake 3.4.1+. You can install Kompute from thePython pypi packageusing the following command.pip install kpYou can also install from master branch using:pip install git+git://github.com/KomputeProject/kompute.git@masterFor further details you can read thePython Package documentationor thePython Class Reference documentation.C++ Build OverviewThe build system provided usescmake, which allows for cross platform builds.The top levelMakefileprovides a set of optimized configurations for development as well as the docker image build, but you can start a build with the following command:cmake -BbuildYou also are able to add Kompute in your repo withadd_subdirectory- theAndroid example CMakeLists.txt fileshows how this would be done.For a more advanced overview of the build configuration check out theBuild System Deep Divedocumentation.Kompute DevelopmentWe appreciate PRs and Issues. If you want to contribute try checking the "Good first issue" tag, but even using Kompute and reporting issues is a great contribution!ContributingDev DependenciesTestingGTestDocumentationDoxygen (with Dot)SphynxDevelopmentFollows Mozilla C++ Style Guidehttps://www-archive.mozilla.org/hacking/mozilla-style-guide.htmlUses post-commit hook to run the linter, you can set it up so it runs the linter before commitAll dependencies are defined in vcpkg.jsonUses cmake as build system, and provides a top level makefile with recommended commandUses xxd (or xxd.exe windows 64bit port) to convert shader spirv to header filesUses doxygen and sphinx for documentation and autodocsUses vcpkg for finding the dependencies, it's the recommended set up to retrieve the librariesIf you want to run with debug layers you can add them with theKOMPUTE_ENV_DEBUG_LAYERSparameter as:export KOMPUTE_ENV_DEBUG_LAYERS="VK_LAYER_LUNARG_api_dump"Updating documentationTo update the documentation you will need to:Run the gendoxygen target in the build systemRun the gensphynx target in the build-systemPush to github pages withmake push_docs_to_ghpagesRunning testsRunning the unit tests has been significantly simplified for contributors.The tests run on CPU, and can be triggered using the ACT command line interface (https://github.com/nektos/act) - once you install the command line (And start the Docker daemon) you just have to type:$ act

[Python Tests/python-tests] 🚀  Start image=axsauze/kompute-builder:0.2
[C++ Tests/cpp-tests      ] 🚀  Start image=axsauze/kompute-builder:0.2
[C++ Tests/cpp-tests      ]   🐳  docker run image=axsauze/kompute-builder:0.2 entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Python Tests/python-tests]   🐳  docker run image=axsauze/kompute-builder:0.2 entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
...The repository contains unit tests for the C++ and Python code, and can be found under thetest/andpython/testfolder.The tests are currently run through the CI using Github Actions. It uses the images found indocker-builders/.In order to minimise hardware requirements the tests can run without a GPU, directly in the CPU usingSwiftshader.For more information on how the CI and tests are setup, you can go to theCI, Docker and Tests Sectionin the documentation.MotivationsThis project started after seeing that a lot of new and renowned ML & DL projects like Pytorch, Tensorflow, Alibaba DNN, Tencent NCNN - among others - have either integrated or are looking to integrate the Vulkan SDK to add mobile (and cross-vendor) GPU support.The Vulkan SDK offers a great low level interface that enables for highly specialized optimizations - however it comes at a cost of highly verbose code which requires 500-2000 lines of code to even begin writing application code. This has resulted in each of these projects having to implement the same baseline to abstract the non-compute related features of the Vulkan SDK. This large amount of non-standardised boiler-plate can result in limited knowledge transfer, higher chance of unique framework implementation bugs being introduced, etc.We are currently developing Kompute not to hide the Vulkan SDK interface (as it's incredibly well designed) but to augment it with a direct focus on the Vulkan SDK's GPU computing capabilities.This articleprovides a high level overview of the motivations of Kompute, together with a set of hands on examples that introduce both GPU computing as well as the core Kompute architecture.