Package: kompute
Vulkan KomputeThe General Purpose Vulkan Compute Framework for C++ and Python.Blazing fast, mobile-enabled, asynchronous, and optimized for advanced GPU processing usecases.🔋Documentation💻Blog Post⌨Examples💾Principles & FeaturesSingle headerfor simple import with flexible build-system configurationMulti-language support with C++ as core SDK as well asoptimized Python bindingsAsynchronous & parallel processingsupport through GPU family queuesMobile enabledwith examples in Android studio across several architecturesBYOV:Bring-your-own-Vulkan designto play nice with existing Vulkan applicationsExplicit relationships for GPU and hostmemory ownership and memory managementShort code examplesshowing the core featuresLonger tutorials formachine learning 🤖,mobile development 📱andgame development 🎮.Getting StartedSetupKompute is provided as a single header fileKompute.hpp. Seebuild-system sectionfor configurations available.Your First Kompute (Simple Version)This simple example will show the basics of Kompute through the high level API.Create Kompute Manager with default settings (device 0 and first compute compatible queue)Create and initialise Kompute Tensors through managerRun multiplication operation synchronouslyMap results back from GPU memory to print the resultsView theextended versionormore examples.intmain(){// 1. Create Kompute Manager with default settings (device 0 and first compute compatible queue)kp::Managermgr;// 2. Create and initialise Kompute Tensors through managerautotensorInA=mgr.buildTensor({2.,2.,2.});autotensorInB=mgr.buildTensor({1.,2.,3.});autotensorOut=mgr.buildTensor({0.,0.,0.});// 3. Run multiplication operation synchronouslymgr.evalOpDefault<kp::OpMult<>>({tensorInA,tensorInB,tensorOut})// 4. Map results back from GPU memory to print the resultsmgr.evalOpDefault<kp::OpTensorSyncLocal>({tensorInA,tensorInB,tensorOut})// Prints the output which is Output: { 2, 4, 6 }std::cout<<fmt::format("Output: {}",tensorOut.data())<<std::endl;}Your First Kompute (Extended Version)We will now show thesame example as abovebut leveraging more advanced Kompute features:Create Kompute Manager with explicit device 0 and single queue of familyIndex 2Explicitly create Kompute Tensors without initializing in GPUInitialise the Kompute Tensor in GPU memory and map data into GPURun operation with custom compute shader code asynchronously with explicit dispatch layoutCreate managed sequence to submit batch operations to the CPUMap data back to host by running the sequence of batch operationsViewmore examples.intmain(){// 1. Create Kompute Manager with explicit device 0 and single queue of familyIndex 2kp::Managermgr(0,{2});// 2. Explicitly create Kompute Tensors without initializing in GPUautotensorInA=std::make_shared<kp::Tensor>(kp::Tensor({2.,2.,2.}));autotensorInB=std::make_shared<kp::Tensor>(kp::Tensor({1.,2.,3.}));autotensorOut=std::make_shared<kp::Tensor>(kp::Tensor({0.,0.,0.}));// 3. Initialise the Kompute Tensor in GPU memory and map data into GPUmgr.evalOpDefault<kp::OpTensorCreate>({tensorInA,tensorInB,tensorOut});// 4. Run operation with custom compute shader code asynchronously with explicit dispatch layoutmgr.evalOpAsyncDefault<kp::OpAlgoBase<3,1,1>>({tensorInA,tensorInB,tensorOut},shaderData);// "shaderData" defined is below and can be glsl/spirv string, or path to file// 4.1. Before submitting sequence batch we wait for the async operationmgr.evalOpAwaitDefault();// 5. Create managed sequence to submit batch operations to the CPUstd::shared_ptr<kp::Sequence>sq=mgr.getOrCreateManagedSequence("seq");// 5.1. Explicitly begin recording batch commandssq->begin();// 5.2. Record batch commandssq->record<kp::OpTensorSyncLocal({tensorInA});sq->record<kp::OpTensorSyncLocal({tensorInB});sq->record<kp::OpTensorSyncLocal({tensorOut});// 5.3. Explicitly stop recording batch commandssq->end();// 6. Map data back to host by running the sequence of batch operationssq->eval();// Prints the output which is Output: { 2, 4, 6 }std::cout<<fmt::format("Output: {}",tensorOut.data())<<std::endl;}Your shader can be provided as raw glsl/hlsl string, SPIR-V bytes array (using our CLI), or string path to file containing either. Below are the examples of the valid ways of providing shader.Passing raw GLSL/HLSL stringstaticstd::stringshaderString=(R"(#version 450layout (local_size_x = 1) in;// The input tensors bind index is relative to index in parameter passedlayout(set = 0, binding = 0) buffer bina { float tina[]; };layout(set = 0, binding = 1) buffer binb { float tinb[]; };layout(set = 0, binding = 2) buffer bout { float tout[]; };void main() {uint index = gl_GlobalInvocationID.x;tout[index] = tina[index] * tinb[index];})");staticstd::vector<char>shaderData(shaderString.begin(),shaderString.end());Passing SPIR-V Bytes arrayYou can use the Komputeshader-to-cpp-header CLIto convert your GLSL/HLSL or SPIR-V shader into C++ header file (see documentation link for more info). This is useful if you want your binary to be compiled with all relevant artifacts.staticstd::vector<uint8_t>shaderData={0x03,//... spirv bytes go here)Path to file containing raw glsl/hlsl or SPIRV bytesstaticstd::stringshaderData="path/to/shader.glsl";// Or SPIR-Vstaticstd::stringshaderData="path/to/shader.glsl.spv";More examplesSimple examplesPass shader as raw stringRecord batch commands with a Kompute SequenceRun Asynchronous OperationsRun Parallel Operations Across Multiple GPU QueuesCreate your custom Kompute OperationsImplementing logistic regression from scratchEnd-to-end examplesMachine Learning Logistic Regression ImplementationParallelizing GPU-intensive Workloads via Multi-Queue OperationsAndroid NDK Mobile Kompute ML ApplicationGame Development Kompute ML in Godot EngineArchitectural OverviewThe core architecture of Kompute includes the following:Kompute Manager- Base orchestrator which creates and manages device and child componentsKompute Sequence- Container of operations that can be sent to GPU as batchKompute Operation (Base)- Base class from which all operations inheritKompute Tensor- Tensor structured data used in GPU operationsKompute Algorithm- Abstraction for (shader) code executed in the GPUTo see a full breakdown you can read further in theC++ Class Reference.Full Vulkan ComponentsSimplified Kompute Components(very tiny, check thefull reference diagram in docs for details)Asynchronous and Parallel OperationsKompute provides flexibility to run operations in an asynrchonous way through Vulkan Fences. Furthermore, Kompute enables for explicit allocation of queues, which allow for parallel execution of operations across queue families.The image below provides an intuition on how Kompute Sequences can be allocated to different queues to enable parallel execution based on hardware. You can see thehands on example, as well as thedetailed documentation pagedescribing how it would work using an NVIDIA 1650 as an example.Mobile EnabledKompute has been optimized to work in mobile environments. Thebuild systemenables for dynamic loading of the Vulkan shared library for Android environments, together with a workingAndroid NDK Vulkan wrapperfor the CPP headers.For a full deep dive you can read the blog post "Supercharging your Mobile Apps with On-Device GPU Accelerated Machine Learning".You can also access theend-to-end example codein the repository, which can be run using android studio.Python PackageBesides the C++ core SDK you can also use the Python package of Kompute, which exposes the same core functionality, and supports interoperability with Python objects like Lists, Numpy Arrays, etc.You can install from the repository by running:pip install .For further details you can read thePython Package documentationor thePython Class Reference documentation.Python Example (Simple)Then you can interact with it from your interpreter. Below is the same sample as above "Your First Kompute (Simple Version)" but in Python:mgr=Manager()# Can be initialized with List[] or np.Arraytensor_in_a=Tensor([2,2,2])tensor_in_b=Tensor([1,2,3])tensor_out=Tensor([0,0,0])mgr.eval_tensor_create_def([tensor_in_a,tensor_in_b,tensor_out])shaderFilePath="shaders/glsl/opmult.comp"mgr.eval_async_algo_file_def([tensor_in_a,tensor_in_b,tensor_out],shaderFilePath)# Alternatively can pass raw string/bytes:# shaderFileData = """ shader code here... """# mgr.eval_algo_data_def([tensor_in_a, tensor_in_b, tensor_out], list(shaderFileData))mgr.eval_await_def()mgr.eval_tensor_sync_local_def([tensor_out])asserttensor_out.data()==[2.0,4.0,6.0]Python Example (Extended)Similarly you can find the same extended example as above:mgr=Manager(0,[2])# Can be initialized with List[] or np.Arraytensor_in_a=Tensor([2,2,2])tensor_in_b=Tensor([1,2,3])tensor_out=Tensor([0,0,0])shaderFilePath="../../shaders/glsl/opmult.comp"mgr.eval_tensor_create_def([tensor_in_a,tensor_in_b,tensor_out])seq=mgr.create_sequence("op")mgr.eval_async_algo_file_def([tensor_in_a,tensor_in_b,tensor_out],shaderFilePath)mgr.eval_await_def()seq.begin()seq.record_tensor_sync_local([tensor_in_a])seq.record_tensor_sync_local([tensor_in_b])seq.record_tensor_sync_local([tensor_out])seq.end()seq.eval()asserttensor_out.data()==[2.0,4.0,6.0]For further details you can read thePython Package documentationor thePython Class Reference documentation.Build OverviewThe build system provided usescmake, which allows for cross platform builds.The top levelMakefileprovides a set of optimized configurations for development as well as the docker image build, but you can start a build with the following command:cmake -BbuildYou also are able to add Kompute in your repo withadd_subdirectory- theAndroid example CMakeLists.txt fileshows how this would be done.For a more advanced overview of the build configuration check out theBuild System Deep Divedocumentation.Kompute DevelopmentWe appreciate PRs and Issues. If you want to contribute try checking the "Good first issue" tag, but even using Vulkan Kompute and reporting issues is a great contribution!ContributingDev DependenciesTestingGTestDocumentationDoxygen (with Dot)SphynxDevelopmentFollows Mozilla C++ Style Guidehttps://www-archive.mozilla.org/hacking/mozilla-style-guide.htmlUses post-commit hook to run the linter, you can set it up so it runs the linter before commitAll dependencies are defined in vcpkg.jsonUses cmake as build system, and provides a top level makefile with recommended commandUses xxd (or xxd.exe windows 64bit port) to convert shader spirv to header filesUses doxygen and sphinx for documentation and autodocsUses vcpkg for finding the dependencies, it's the recommended set up to retrieve the librariesUpdating documentationTo update the documentation you will need to:Run the gendoxygen target in the build systemRun the gensphynx target in the build-systemPush to github pages withmake push_docs_to_ghpagesRunning testsTo run tests you can use the helper top level MakefileFor visual studio you can runmake vs_cmake
make vs_run_tests VS_BUILD_TYPE="Release"For unix you can runmake mk_cmake MK_BUILD_TYPE="Release"
make mk_run_testsMotivationsThis project started after seeing that a lot of new and renowned ML & DL projects like Pytorch, Tensorflow, Alibaba DNN, Tencent NCNN - among others - have either integrated or are looking to integrate the Vulkan SDK to add mobile (and cross-vendor) GPU support.The Vulkan SDK offers a great low level interface that enables for highly specialized optimizations - however it comes at a cost of highly verbose code which requires 500-2000 lines of code to even begin writing application code. This has resulted in each of these projects having to implement the same baseline to abstract the non-compute related features of Vulkan. This large amount of non-standardised boiler-plate can result in limited knowledge transfer, higher chance of unique framework implementation bugs being introduced, etc.We are currently developing Vulkan Kompute not to hide the Vulkan SDK interface (as it's incredibly well designed) but to augment it with a direct focus on Vulkan's GPU computing capabilities.This articleprovides a high level overview of the motivations of Kompute, together with a set of hands on examples that introduce both GPU computing as well as the core Vulkan Kompute architecture.